Eval-driven development: building an evaluation pipeline for AI features
A test suite exists because nobody accepts "it looked right when I tried it" as a release gate for ordinary code. A prompt or an agent's tool configuration can regress silently just as a function can. Eval-driven development treats a change to an AI feature like a code change: a regression suite runs before it ships, so nobody has to notice the breakage in production.
1. The gap this closes
AI evaluation methods covers the mechanics (rubrics, LLM-as-judge, benchmarks) of measuring whether an output is good. Eval-driven development is about when and how those measurements run. They run as a standing pipeline, automatically, against every change to a prompt, a tool configuration, or a model version, the way a test suite runs against every code change however confident the author is. It closes the gap between "I tried it on a few examples and it looked fine" and "I ran it against a representative dataset and can see exactly what changed." A test suite closes the same gap for ordinary code. Here it's applied to a system whose correctness is harder to check by eye.
2. A golden dataset, built from real examples
The eval dataset has to be made of real, representative inputs: past queries, documents the system has processed, edge cases that showed up in production. Examples someone invented to have something to test against don't qualify. An invented dataset reflects what its author assumed the hard cases would be, which is narrower than what usage produces. A system that passes a hand-invented set can still fail badly on live traffic, because the set never included the weird, ambiguous, edge-case-heavy inputs users reliably send.
Build the dataset from logged production inputs, sampled to cover both the common case and the tail. Label each with a correct or acceptable answer, ideally from a reviewer with domain expertise in what "correct" means for this system. Treat building it as an ongoing engineering task, not a bootstrap exercise done once before launch.
3. Wiring evals into the same pipeline as code review
The discipline sticks under deadline pressure only if the eval run triggers automatically on the same event as code review. A prompt change, a system-prompt edit, a tool-schema update, or a model version bump submitted as a pull request runs the eval suite the way a code change runs tests. Reviewers see the results before the change merges. Nobody has to remember to check.
This also makes a prompt change reviewable like a code change. A reviewer sees the diff to the prompt text and the diff to the eval output. That is a far more concrete basis for approving or pushing back than reading the new wording and guessing how it will behave across the inputs the system handles.
4. Regression detection: compare to baseline, not to an absolute bar
An eval score alone ("this prompt scores 82 on the rubric") tells you less than it seems. Without a baseline, 82 could be an improvement or a regression depending on what the previous version scored. Always compare: does this change score better, worse, or the same as the previous version against the same dataset, and which individual examples flipped from passing to failing, or back?
The per-example view matters because a flat aggregate can hide a regression. A change that fixes five failing examples and breaks five different passing ones nets the same aggregate as no change. A reviewer looking only at the top-line number would approve a change that traded one class of correctness for another, which is a bad trade if the newly broken examples matter more. Show the specific examples that changed status, not only the aggregate delta, so the reviewer can make that call.
5. The eval set itself decays
A golden dataset built from production traffic reflects the moment it was sampled, and production drifts. User behavior changes, new features shift what a typical input looks like, an upstream data source changes format. An eval set that was representative a year ago can stop being representative without anyone deciding it, the way alert thresholds tuned against an old baseline stop making sense once normal behavior moves.
Give eval-set maintenance an owner and a schedule. Re-sample from current production traffic. Check whether existing labels still reflect what "correct" means for the system as it has evolved. Retire examples for cases the system no longer needs to handle. A suite that passes cleanly against a stale dataset gives false confidence: it measures a distribution the system barely sees while the current failure modes go unmeasured.
6. Human review as an ongoing part of the loop, not a bootstrap step
It's tempting to treat human labeling as a one-time cost: label the initial dataset, then run the automated suite indefinitely. The automated suite itself needs periodic human spot-checks, especially the LLM-as-judge portion AI evaluation methods covers, to catch drift in what the judge accepts. That guide describes the same calibration risk for judge models generally. A reviewer periodically samples a subset of the automated verdicts to confirm the scoring still agrees with human judgment. No full relabel is needed. Without this, the pipeline drifts into measuring something slightly different from what it was built to measure.
7. Worked example: a change that passed manual review and failed the eval suite
A team updates a support bot's system prompt to handle a new product category. The wording reads correctly on the handful of manual spot-checks the author tries before opening a pull request. The eval suite, run automatically on the pull request, finds a regression the spot-checks missed: several passing examples about an older product category now fail. The new instructions shifted how the model interprets an ambiguous phrase that also appears in the older category's typical queries, and none of the manual test cases included it.
Without the eval suite, this ships. The regression surfaces later in production, harder to trace once other changes have shipped on top. With the suite in the review pipeline, the regression shows up in the pull request itself, tied to the exact examples that flipped. The author gets a concrete, reproducible case to fix the prompt against, in place of a vague report days later that support quality for an unrelated category got worse.
This pairs with AI evaluation methods for the measurement techniques an eval pipeline runs, and with context engineering for why a prompt or context-structure change is the kind of edit whose effects are hardest to predict from the diff. That unpredictability is the argument for running it against a dataset instead of trusting a read-through.