Every team shipping an LLM feature goes through the same phase. You write a prompt, try it on three or four inputs you thought of, read the outputs, and they look… good. You tweak a sentence, try the same inputs again, and they still look good. So you ship it.

Two weeks later a user pastes in a support ticket with a table in it, and the feature confidently summarises a refund policy that doesn’t exist.

Nothing went wrong in that story except the evaluation. “Looks good to me” is a vibe check, and vibe checks have three fatal properties. They use inputs you chose, so they’re biased toward what you already handle. They’re graded by the person who wrote the prompt, so they’re biased toward passing. And they leave no record, so next week’s change can’t be compared against this week’s. Evaluation is how you stop grading your own homework.

This post is the process I now use for every LLM feature — and, more usefully, the specific ways I’ve watched that process lie to me.

Start from failures, not from a metric

The instinct is to reach for a metric first: pick “faithfulness” or “helpfulness” off a framework’s menu, wire up a score, watch the dashboard. I’ve learned to do that last. Generic metrics measure generic things. Your feature fails in specific ways, and you don’t know what they are yet.

So the first eval step isn’t code. It’s reading. I pull 50–100 real traces — or, before launch, outputs on realistic inputs gathered from teammates — and read every one end to end, writing a one-line note on anything wrong. No categories up front, just notes. “Ignored the date the user gave.” “Summarised the quoted email instead of the reply.” “Correct, but four paragraphs for a yes/no question.”

After a few dozen, the notes cluster on their own. That clustering is your failure taxonomy, and it’s worth more than any off-the-shelf metric because every category is something that actually happened. For one document-extraction feature, the taxonomy came out as:

  • Hallucinated fields — values in the output that appear nowhere in the source.
  • Wrong entity — the right field, pulled from the wrong party in the document.
  • Format drift — dates and currencies in inconsistent shapes.
  • Over-refusal — “I can’t determine this” when the answer was on page one.

Four categories, ranked by frequency and by cost. That list tells you what to measure and where to spend prompt-engineering time. Teams that skip this step end up with an elaborate eval suite that checks things that were never broken.

If you haven’t read your traces, your metrics are measuring your assumptions.

Build a golden set that looks like reality

A golden set is a versioned collection of inputs, each paired with an expected output or a description of what a good output must and must not do. It’s the unit-test suite of an LLM feature — with the difference that you’ll never reach 100%, and shouldn’t expect to.

Where the cases come from matters more than how many there are:

  • Sampled real traffic — the bulk of the set. Stratify by the slices you care about (language, input length, customer tier, document type) so common cases can’t drown out rare ones.
  • Every production incident — each bug that reaches a user becomes a case, permanently. This is the ratchet that makes the suite better over time.
  • Deliberate edge cases — empty inputs, enormous inputs, prompt injections hidden in user content, the adversarial traffic you know is coming.

I start with a few dozen cases and grow the set every week. A couple of hundred well-chosen examples will catch more than ten thousand synthetic ones. Tag every case with its slice and failure category, so a report can say “hallucinated fields went from 4% to 11% on scanned PDFs” instead of “the score dropped two points.”

One rule I hold firmly: split the set. Keep a development portion you iterate against freely, and a held-out portion you only run before a release. Tune prompts against every case you have and you will overfit to them — slowly, invisibly, one small tweak at a time — while your score climbs and real quality stays flat.

Use the cheapest grader that works

Once you know what to check, you need something to check it. I think of graders as a ladder, and always start on the bottom rung:

  1. Deterministic assertions. Does it parse against the schema? Is the date ISO-8601? Is it under the length budget? Does it leak the system prompt? These checks are free, instant and perfectly reproducible, and they cover more of your taxonomy than you’d expect.
  2. Reference comparisons. For tasks with a right answer — classification, extraction, routing — compare against the labelled answer field by field: exact match where it’s exact, normalised match (case, whitespace, number formats) where it isn’t.
  3. LLM-as-judge. For qualities code can’t check — is this summary faithful to the source, does this reply actually answer the question — use a model to grade. Powerful, but now you have a grader you need to evaluate too.
  4. Human review. The ground truth that calibrates everything above it, and the only grader to trust for genuinely subjective quality.

A harness covering the first two rungs is small:

type Case = { id: string; input: string; expected?: Extracted; tags: string[] };

async function runSuite(cases: Case[], pipeline: Pipeline) {
  const results = [];
  for (const c of cases) {
    const out = await pipeline.run(c.input);
    const checks = {
      parses: Schema.safeParse(out.data).success,
      noLeak: !out.raw.includes(SYSTEM_PROMPT_MARKER),
      fields: c.expected ? fieldAccuracy(out.data, c.expected) : null,
    };
    results.push({ id: c.id, tags: c.tags, checks });
  }
  return summarizeBySlice(results); // pass rate per tag, never one global number
}

Making an LLM judge trustworthy

LLM judges fail in predictable ways, and the fixes are just as predictable:

  • Ask for a binary verdict, not a 1–10 score. A judge asked for a score clusters around 7 and drifts between runs. A judge asked “does the summary make any claim the source doesn’t support — yes or no?” is far more stable. If you need nuance, split it into several binary questions.
  • Make it show evidence before it rules. Have the judge quote the supporting text, then give the verdict. Accuracy goes up, and every failure comes with an audit trail.
  • Judge one criterion at a time. A single prompt grading faithfulness, tone and completeness at once does all three badly.
  • Calibrate it against humans. Hand-label a hundred or so outputs, run the judge on the same set, and measure agreement separately for passes and for failures. A judge that agrees 95% of the time but misses half the real failures is worse than useless — it’s confidently wrong in exactly the direction you care about.
You are checking a summary against its source document.

<source>{source}</source>
<summary>{summary}</summary>

List every factual claim in the summary. For each one, quote the sentence
in the source that supports it, or write NONE.

On the final line, answer exactly PASS if every claim is supported,
or FAIL if any claim is marked NONE.

Re-run the calibration whenever you change the judge’s prompt or model. The judge is code, and it regresses like code.

Six ways your evals will lie to you

This is the part tutorials skip. A harness hands you a number, and numbers feel objective. These are the ways that number has misled me.

1. Noise that looks like signal. With 50 cases, a single flip moves your score by two points, and LLM outputs aren’t deterministic even at temperature 0 — so a “two-point regression” can appear from re-running the exact same prompt. Before believing a delta, run the baseline a few times and see how much it wobbles on its own. For decisions that matter, compare candidate and baseline case by case: ten fixed and nine broken isn’t an improvement, it’s churn.

// Paired comparison: what actually changed, case by case?
const fixed  = cases.filter(c => !base[c.id].pass &&  cand[c.id].pass);
const broken = cases.filter(c =>  base[c.id].pass && !cand[c.id].pass);
// Read every broken case by hand before shipping, whatever the headline says.

2. Averages that hide regressions. An overall pass rate up 3% can conceal a slice that fell off a cliff: the new prompt is better at English, worse at German, and there’s more English. Report per slice, and gate on the worst slice rather than the mean.

3. Overfitting the golden set. It feels like progress. If your dev-set score keeps climbing while the held-out score doesn’t, you’re memorising the test.

4. A judge that shares your blind spots. Using the same model to generate and to judge invites it to approve of its own style. Where you can, judge with a different — usually stronger — model, and keep a human spot-checking a sample of verdicts every release.

5. A pinned model that isn’t pinned. Provider-side updates change behaviour under an unchanged prompt. Pin exact model versions in production, record the version in every eval run, and treat a model upgrade as a change that has to pass the suite like any other.

6. Measuring the wrong thing. The most insidious one. The suite says 94%; users are unhappy. That gap means your taxonomy has gone stale, and the fix is to go back to reading traces. An eval is a model of what users care about — and models drift.

Wire it into the release loop

An eval you run by hand when you remember is a vibe check with extra steps. The value comes from making it automatic, and making it block.

OBSERVECATEGORISECAPTUREBLOCKRELEASEprod traceserror analysisgolden setCI eval gateshipnew failures in production become new cases
The eval flywheel — every failure that escapes to production comes back as a case the suite checks forever.

In practice, that’s three tiers:

  • On every pull request: the deterministic checks plus a fast, representative slice of the golden set. A few minutes, cheap enough to run on every prompt edit.
  • Nightly and before every release: the full suite, including the held-out split and the LLM-judged criteria, with a few repeats per case to smooth out noise.
  • The gate itself: a release is blocked if any tagged slice drops beyond its noise band, or if any case tagged as a past incident fails. Past incidents are non-negotiable — that’s the whole point of keeping them.

Keep the results, not just the pass/fail. A history of scores per slice, per model version and per prompt version is what turns “when did this start?” into a five-minute question instead of a two-day one.

Keep evaluating after you ship

Offline evals tell you whether a change is safe to ship. They can’t tell you how the feature behaves on inputs nobody imagined. For that you need online signals:

  • Implicit feedback — users editing the output, regenerating it, copying it, or abandoning the flow. Noisier than a thumbs-up button, but orders of magnitude more plentiful.
  • Sampled judging on live traffic — run your calibrated judges on a small percentage of production outputs, and alert when a failure category spikes.
  • Guardrail metrics — refusal rate, output length, latency and cost per request. A sudden shift in any of them is often the first sign that behaviour has changed.

Then close the loop: the interesting production failures go straight back into the error analysis at the top of this post. That’s the flywheel. With each lap, the suite looks a little more like reality.

Before you call an LLM feature done, check:

  • You’ve read real outputs and written down how it fails, not just whether.
  • There’s a versioned golden set, tagged by slice, with a held-out split you never tune against.
  • Cheap deterministic checks run first; LLM judges are binary, single-criterion and calibrated against human labels.
  • Every comparison is paired and per-slice, and you know how much your baseline wobbles on its own.
  • Evals run in CI and block releases, and past incidents can never silently regress.
  • Production is sampled, judged and fed back into the set.

Evals won’t make a feature good. They’ll tell you — quickly and honestly — whether it is, which is the one thing a vibe check never could.