How to Build an LLM Evaluation Set That Is Worth Trusting

If we could enforce one practice on every AI project, it would be this one. An evaluation set is what turns “this feels better” into a number, and without it a system cannot be improved deliberately — only changed and hoped about.

Here is the method we use, which takes roughly a week.

Step 1: Take the cases from your own history

Your company already has a labelled dataset. It is just not formatted as one.

Support transcripts with their resolutions. Tickets that were closed. Applications that were approved or declined. Documents that were coded. Cases that were adjudicated. Each of these is an input with a known outcome, produced by a human who was accountable for it.

Pull from that. Specifically:

• Sample to match your real distribution. If 40% of your tickets are order status, 40% of your evaluation set should be too. A set weighted toward interesting cases produces numbers that do not predict production behaviour.
• Include the tail. The awkward, ambiguous, multi-issue cases are where systems fail, and they are exactly what a hand-picked set omits.
• Include cases a human got wrong, labelled with what the right answer was. These are disproportionately valuable, because they are the cases where the process itself is hard.
• Include cases with no answer. Questions your corpus does not cover, requests outside scope. Refusal has to be scored as a correct outcome or the system learns to always answer.

Step 2: Adjudicate with someone who knows the domain

An engineer cannot decide what the correct outcome is for a clinical coding question or a credit exception. Sit with the person who does this work and agree, case by case, what a correct response looks like.

Two things reliably happen during this exercise, and both are valuable beyond the evaluation set:

You discover cases where two experienced people disagree. Those are not evaluation cases — they are a signal that the process has no defined correct answer, and automating them means automating inconsistency. We have recommended excluding whole categories on this basis.

You discover the criteria that were never written down. Someone explains why this case resolves differently from that one, and you are hearing policy that exists only in practice.

Step 3: Break the scores out by category

A single aggregate accuracy number is close to useless. It hides the thing you need to see.

If your set has eight categories and one of them regresses badly while the others improve slightly, the average moves a fraction of a point and you learn nothing. Report per category, and set thresholds per category.

Step 4: Pick the scoring method per case type

Not everything needs the same scorer:

• Structured extraction — exact match or schema validation. Deterministic, cheap, unambiguous.
• Retrieval — recall and precision at k, scored against the passages that genuinely contain the answer. Measured independently of the final answer.
• Refusal — did the system correctly decline when it should have.
• Open-ended text — a model-as-judge against a rubric, which needs the next step.

Step 5: Validate the judge before trusting it

This is the step most often skipped, and skipping it invalidates everything downstream.

An unvalidated judge produces numbers that look like measurement and correlate with nothing. Before relying on one:

1. Have humans label a sample of outputs against the same rubric.
2. Measure the judge’s agreement with those labels.
3. If agreement is poor, the rubric is ambiguous — rewrite it and repeat.

When the judge disagrees with your domain experts, the rubric is wrong, not the experts. Fix it until agreement is high enough that you would act on the judge’s verdict.

Step 6: Gate the deploy with it

Evaluation that runs manually gets skipped under deadline pressure. Make it run on every pull request and block the merge when a category score drops past threshold.

This is not a novel practice. It is how you already treat unit tests. The only reason AI changes are commonly exempt is that the tooling arrived later.

Step 7: Keep feeding it from production

The set reflects the traffic that existed when you built it. Six months on, real usage has moved.

Sample live traffic continuously, score it, and feed the interesting cases — especially the failures — back into the set. An evaluation suite nobody maintains stops being true within a quarter, which is why we insist on naming an owner for it before handover.

What this costs and what it returns

A week of work, mostly adjudication time from a domain expert.

What it returns: the ability to compare two prompts, evaluate a new model version in an afternoon, detect a regression before a customer does, give a compliance reviewer a number instead of a reassurance, and improve the system deliberately rather than by trial and error. It is the cheapest leverage available in an AI project.