Prompt library + evals

AI Enablement · Initiative playbook
Back to Growth Engineering
AI Enablement

Prompt library + evals

Teams copy-paste prompts from Slack with no versioning, no quality bar, and no rollback path.

The problem

Teams copy-paste prompts from Slack with no versioning, no quality bar, and no rollback path.

The play

Versioned prompt registry with regression evals on every change. Each workflow tracks deflection rate and quality score, so adoption is safe by default.

Headline KPI

Prompt regressions caught pre-prod
0%≥90%
BAU owner
AI Platform

Tools in the stack

LangSmithGitHubClaude

Rollout plan

  1. 1
    Discover
    Audit the current process, instrument baseline metrics, and confirm the bottleneck with the BAU owner.
  2. 2
    Design
    Versioned prompt registry with regression evals on every change. Each workflow tracks deflection rate and quality score, so adoption is safe by default.
  3. 3
    Pilot
    Run with Support macro generator. Measure against the success criteria below; iterate weekly.
  4. 4
    Evaluate
    Review against: Zero prod regressions over 30 days. Decide go / no-go with AI Platform.
  5. 5
    Embed in BAU
    Hand over to AI Platform with runbook, dashboards, and on-call rotation. Schedule 60-day ROI check.

Success criteria

Pilot scope
Support macro generator
Definition of done
Zero prod regressions over 30 days
BAU owner
AI Platform