--- title: "How to Build an LLM Evaluation Set That Is Worth Trusting" section: "Insights" canonical_url: "https://leverge.ai/insights/building-an-llm-evaluation-set" topic: "how to build an LLM evaluation set" published: "2026-07-16" updated: "2026-07-31" publisher: "Ailoitte Technologies Private Limited" --- # How to Build an LLM Evaluation Set That Is Worth Trusting A useful evaluation set comes from your own history, not from synthetic examples. The method is: sample real cases to match your actual distribution, adjudicate correct outcomes with someone who knows the domain, break scores out by category rather than reporting one average, pick a scoring method per case type, and validate any model-as-judge against human labels before trusting it. Two to three hundred cases is usually enough to start, and the whole exercise takes about a week. ## Key takeaways - Cases must come from your real history, because synthetic examples only test the situations you already thought of. - Two to three hundred well-chosen cases beat two thousand generated ones, and can be adjudicated in about a week. - Report scores per category rather than as one average, because an average hides exactly the regression you need to see. - A model-as-judge must be validated against human labels first, or you are measuring one model's opinion of another. - Include cases the corpus does not cover and score refusal as a correct outcome, otherwise the system learns to always answer. If we could enforce one practice on every AI project, it would be this one. An evaluation set is what turns "this feels better" into a number, and without it a system cannot be improved deliberately — only changed and hoped about. Here is the method we use, which takes roughly a week. ## Step 1: Take the cases from your own history Your company already has a labelled dataset. It is just not formatted as one. Support transcripts with their resolutions. Tickets that were closed. Applications that were approved or declined. Documents that were coded. Cases that were adjudicated. Each of these is an input with a known outcome, produced by a human who was accountable for it. Pull from that. Specifically: - **Sample to match your real distribution.** If 40% of your tickets are order status, 40% of your evaluation set should be too. A set weighted toward interesting cases produces numbers that do not predict production behaviour. - **Include the tail.** The awkward, ambiguous, multi-issue cases are where systems fail, and they are exactly what a hand-picked set omits. - **Include cases a human got wrong**, labelled with what the right answer was. These are disproportionately valuable, because they are the cases where the process itself is hard. - **Include cases with no answer.** Questions your corpus does not cover, requests outside scope. Refusal has to be scored as a correct outcome or the system learns to always answer. ## Step 2: Adjudicate with someone who knows the domain An engineer cannot decide what the correct outcome is for a clinical coding question or a credit exception. Sit with the person who does this work and agree, case by case, what a correct response looks like. Two things reliably happen during this exercise, and both are valuable beyond the evaluation set: You discover cases where two experienced people disagree. Those are not evaluation cases — they are a signal that the process has no defined correct answer, and automating them means automating inconsistency. We have recommended excluding whole categories on this basis. You discover the criteria that were never written down. Someone explains why this case resolves differently from that one, and you are hearing policy that exists only in practice. ## Step 3: Break the scores out by category A single aggregate accuracy number is close to useless. It hides the thing you need to see. If your set has eight categories and one of them regresses badly while the others improve slightly, the average moves a fraction of a point and you learn nothing. Report per category, and set thresholds per category. ## Step 4: Pick the scoring method per case type Not everything needs the same scorer: - **Structured extraction** — exact match or schema validation. Deterministic, cheap, unambiguous. - **Retrieval** — recall and precision at k, scored against the passages that genuinely contain the answer. Measured independently of the final answer. - **Refusal** — did the system correctly decline when it should have. - **Open-ended text** — a model-as-judge against a rubric, which needs the next step. ## Step 5: Validate the judge before trusting it This is the step most often skipped, and skipping it invalidates everything downstream. An unvalidated judge produces numbers that look like measurement and correlate with nothing. Before relying on one: 1. Have humans label a sample of outputs against the same rubric. 2. Measure the judge's agreement with those labels. 3. If agreement is poor, the rubric is ambiguous — rewrite it and repeat. When the judge disagrees with your domain experts, the rubric is wrong, not the experts. Fix it until agreement is high enough that you would act on the judge's verdict. ## Step 6: Gate the deploy with it Evaluation that runs manually gets skipped under deadline pressure. Make it run on every pull request and block the merge when a category score drops past threshold. This is not a novel practice. It is how you already treat unit tests. The only reason AI changes are commonly exempt is that the tooling arrived later. ## Step 7: Keep feeding it from production The set reflects the traffic that existed when you built it. Six months on, real usage has moved. Sample live traffic continuously, score it, and feed the interesting cases — especially the failures — back into the set. An evaluation suite nobody maintains stops being true within a quarter, which is why we insist on naming an owner for it before handover. ## What this costs and what it returns A week of work, mostly adjudication time from a domain expert. What it returns: the ability to compare two prompts, evaluate a new model version in an afternoon, detect a regression before a customer does, give a compliance reviewer a number instead of a reassurance, and improve the system deliberately rather than by trial and error. It is the cheapest leverage available in an AI project. ## Frequently asked questions ### How many evaluation cases do we need? Two to three hundred is usually enough to start, provided they are sampled to match your real distribution and cover your known-hard categories. Volume matters less than representativeness — two thousand synthetic cases that all look like your happy path tell you less than two hundred real ones that include the tail. Grow the set over time from production traffic rather than trying to build it comprehensively up front. ### Can we use synthetic data for evaluation? As a supplement, for coverage of rare paths your history is too thin on. Not as the foundation. Synthetic cases test the situations you thought of, and production failures come from the situations you did not — which is precisely the gap the evaluation set exists to close. ### How do we validate a model-as-judge? Have humans label a sample of outputs against the same rubric, then measure the judge's agreement with those labels. If agreement is low, the rubric is ambiguous or wrong, and you iterate on it until agreement is high enough to be useful. Skipping this step produces confident numbers that correlate with nothing, which is worse than having no score because it feels like measurement. ### How often should the evaluation set be updated? Continuously, from sampled production traffic. A set built today reflects today's usage; six months later real traffic has drifted — new question types, changed documents, different user behaviour. Feeding sampled live cases back into the set is what keeps the numbers meaningful rather than historical. --- Source: https://leverge.ai/insights/building-an-llm-evaluation-set — Leverge