Evaluation and Testing

1. Why evaluation is the missing discipline

There is a version of software engineering in which correctness is defined by a test suite. You write a function, you write tests against a specification, and the tests either pass or fail. The quality of the function is not a matter of opinion; it is a matter of evidence. Prompt engineering, practised without evals, has no equivalent of this. The prompt either "feels right" or it does not — and that is not a foundation you can build a reliable system on.

The core difficulty is that language model outputs are stochastic and high-dimensional. The same prompt can produce different outputs on successive runs. The output space is not a binary pass/fail but a continuous distribution over quality. This means evaluation for LLM systems is inherently probabilistic: you are not asking "does this return the right value?" but "does this system produce outputs in the acceptable quality band, reliably, across a representative distribution of inputs?"

Shankar et al. (2024) demonstrate one dimension of this in their study of how teams calibrate LLM-assisted evaluation against human preferences — showing systematic divergence when judge pipelines are not aligned.^#Validator The eval infrastructure is not a luxury that comes after the product is working. It is the instrument that tells you whether the product is working in the first place.

In practice, teams without structured eval pipelines often overestimate system quality, misattribute failures to the wrong components, and regress on previously working capabilities when iterating on prompts without regression tests. These are recurring practitioner failure patterns to design against.

A secondary reason evaluation is neglected: it requires making the quality bar explicit. Saying "the output should be helpful and accurate" is not a specification — it is a wish. Writing an eval forces you to operationalise quality: accurate with respect to what ground truth, helpful by what scoring rubric, evaluated against which input distribution. That exercise, uncomfortable as it is, surfaces disagreements about what the system is actually supposed to do before those disagreements surface in production.

2. The evaluation stack

A mature evaluation framework has three layers, each answering a different question. Running only the top layer — end-to-end human evaluation — is expensive, slow, and produces no signal about where a failure originates. Running only the bottom layer — unit tests on individual prompts — tells you nothing about whether the full pipeline works. You need all three.

In practice, L1 and L2 run automatically on every prompt change — they are your CI pipeline. L3 runs periodically (weekly, or before a major release) and when automated scores change in unexpected ways. The outputs of L3 are used to calibrate L1 and L2: if a judge prompt scores outputs highly that human raters consistently score poorly, the judge prompt needs to be revised.

The cost pyramid is inverted: L1 is cheap enough to run on every commit, L2 costs tens to hundreds of API calls per run, L3 costs human time. Design your pipeline so that L1 catches the majority of regressions before they reach L2, and L2 catches failures before they reach L3.

3. Building the eval dataset

An eval dataset is a collection of (input, expected output, metadata) triples against which your system is measured. Building it well is one of the highest-leverage investments in the entire prompt engineering workflow — a dataset that does not represent your true input distribution will produce eval scores that are meaningless, or worse, misleading.

The golden set

Start with a golden set of 50 to 100 manually verified examples. This is the minimum viable eval — small enough to annotate carefully, large enough to detect meaningful performance differences between prompt versions. The examples should be drawn from your actual or expected input distribution, not from convenient cases you can easily verify.

A well-constructed golden set has four components for each example: the input, the ground-truth answer, a rubric that defines what a correct response looks like, and a category label that lets you analyse performance by input type. The category label is particularly important — an aggregate score of 82% hides whether your system is failing uniformly or failing catastrophically on a specific subcategory.

Adversarial and edge cases

A golden set drawn only from representative inputs will miss the failure modes that matter most. Dedicate at least 20% of your eval set to adversarial and edge cases: inputs designed to probe the boundaries of the system's reliability.

As the system matures, the most valuable additions to the eval set are failure-derived cases: real inputs from production where the system produced a wrong or problematic output. These are the cases you did not anticipate when you designed the system, which makes them the cases the system is most likely to continue failing on. Every production failure should produce at least one new eval case.

4. Metrics taxonomy

No single metric captures output quality across all tasks. The right choice depends on the task type, the output format, and how much you can invest in annotation. The following three tiers cover the full range of LLM evaluation needs, from the cheapest to the most expressive.

Exact match and rule-based metrics

Exact string match, regex matching, and schema validation are the cheapest and most reliable metrics available — when they are applicable. They are applicable when the output format is fully constrained: the model is expected to return a specific JSON schema, a classification label from a fixed vocabulary, a boolean, or a number. In these cases, a rule-based check is both faster and more reliable than any model-based metric.

The failure mode is obvious: exact match is too strict for any task involving natural language generation. A response that correctly answers a question in different words will fail an exact-match evaluation. Use rule-based metrics for output structure and factual keys (does the JSON contain the required fields? is the label in the allowed set?), not for the content of generated text.

Semantic similarity

Reference-based metrics compare the generated output against a reference string using embeddings or n-gram overlap. ROUGE (Recall-Oriented Understudy for Gisting Evaluation) measures n-gram overlap and is the standard for summarisation tasks.^#ROUGE BERTScore computes token-level cosine similarity between the generated output and reference using contextual embeddings, which makes it more robust to paraphrase than n-gram methods.^#BERTScore

The limitation of all reference-based metrics is that they require a reference. When multiple correct answers exist, or when the reference was written by a human who used particular phrasing, the score penalises correct paraphrases and rewards output that superficially resembles the reference regardless of factual correctness. Treat these metrics as necessary but not sufficient: high ROUGE does not mean high quality; low ROUGE does not mean low quality.

LLM-as-judge

The most expressive and increasingly dominant approach: use a second language model — typically a stronger model than the one being evaluated — as an automated judge. The judge model reads the input, the system output, and optionally a reference answer, and scores the output according to a rubric you define in the judge prompt.

Zheng et al. (2023) formalise this in the MT-Bench evaluation framework, demonstrating that GPT-4 as a judge achieves over 80% agreement with human expert evaluators on open-ended instruction-following tasks — comparable to the agreement rate between two independent human annotators — across a range of open-ended generation tasks.^#Judge This finding has held across subsequent studies and has established LLM-as-judge as the practical standard for evaluating generated text at scale.

5. Designing the judge prompt

An LLM judge is only as good as its judge prompt. A vague judge prompt produces noisy, inconsistent scores that add variance without adding signal. The design principles below are derived from the MT-Bench methodology and from the practical findings of Shen et al. (2023), who found LLM evaluators inconsistent and dimension-dependent in their evaluation of abstractive summarisation — an early warning that uncalibrated judge pipelines produce unreliable signals.^{#Summarization}

Use a point score with explicit criteria, not a relative comparison. Asking the judge to pick the better of two outputs introduces position bias — the judge systematically favours the output presented first. Absolute scoring against a rubric produces more consistent results and is easier to interpret.

Define each score level explicitly. "Rate on a scale of 1 to 5" is not a rubric. A rubric tells the judge what specifically differentiates a 3 from a 4 — otherwise, score distributions cluster around 3-4 because the judge has no principled reason to use the extremes.

Request a structured response with reasoning first. Forcing the judge to produce an explanation before committing to a score reduces positional anchoring and produces a reasoning trace you can inspect when scores are unexpected. This is the chain-of-thought principle applied to evaluation.

Three known biases to test for and mitigate in your judge prompt. Verbosity bias: judges tend to score longer responses higher regardless of content quality — counteract by explicitly penalising unnecessary length in the rubric. Self-enhancement bias: a model used as its own judge scores its own outputs higher than an independent judge would — use a different model family as the judge. Format bias: responses with markdown formatting, headers, and lists are scored higher than plain prose of equal quality — control for this by stripping or normalising output formatting before evaluation. The specific bias types — verbosity, self-enhancement, and format bias — are widely documented across LLM evaluation research, including in the MT-Bench study above.

Calibrate your judge against human annotations before trusting it at scale. If your judge and your human raters disagree on more than 25% of examples, the judge prompt needs revision — not a stronger model, but a better rubric. Model capability is rarely the bottleneck in judge quality; rubric precision almost always is.

6. Evaluating RAG pipelines

A RAG pipeline has two components that must be evaluated independently: the retrieval system and the generation system. Evaluating only the final output — "did the answer come out right?" — makes it impossible to localise failures. A wrong answer could mean the right chunks were retrieved but the model ignored them, or the wrong chunks were retrieved and the model faithfully reproduced them. The fix is different in each case.

Es et al. (2023) introduce the RAGAS framework — Retrieval-Augmented Generation Assessment — as a largely reference-free evaluation methodology — context recall requires a gold-standard answer, while the other three metrics do not — that decomposes RAG quality into four independent metrics, each targeting a specific failure mode.^#RAGAS

Retrieval metrics

Context recall measures whether the retrieved chunks contain enough information to answer the question. It is computed by checking whether each claim in the ground-truth answer is attributable to at least one retrieved chunk — a reference-based metric that requires a gold-standard answer but makes retrieval quality directly measurable. Low context recall means the correct information was not retrieved; the fix is in the chunking or retrieval configuration, not the generation prompt.

Context precision measures the proportion of retrieved chunks that are actually relevant to the query. A system that retrieves ten chunks, of which only two are relevant, has low precision — and the eight irrelevant chunks are diluting the signal and potentially triggering the over-retrieval failures described in Lesson 05. Context precision is reference-free: it can be computed with an LLM judge that rates each retrieved chunk's relevance to the query independently.

Generation metrics

Faithfulness is the most important RAG-specific generation metric. It measures whether every factual claim in the generated answer is supported by the retrieved context — not whether the answer is correct, but whether it is grounded. A faithful answer based on wrong context is still a faithfulness success; fixing the faithfulness score is a retrieval problem, not a generation problem. Use a judge prompt structured like the example in Section 5.

Answer relevance measures whether the generated answer actually addresses the question asked. A common failure mode in RAG systems is over-hedging: the model produces a response like "Based on the available information, there are several considerations…" that is technically grounded but is not a direct answer to the query. Answer relevance is computed by asking a judge whether the question is fully answered, partially answered, or evaded — and it catches this hedging pattern that faithfulness scores miss entirely.

Run all four metrics on every eval set update. Track them as separate time series. A prompt change that improves faithfulness while degrading context precision, or improves answer relevance while degrading context recall, is telling you something specific about what changed — and that specificity is what makes iteration tractable.

7. Regression testing and CI

A prompt is a software artifact. It should be version-controlled, reviewed, and tested before deployment — the same way any other code change is. The reason most teams do not do this is that the tooling for prompt versioning has historically been poor and the feedback loop for prompt changes has been informal. Both of those conditions have improved substantially. There is no longer a good excuse for deploying prompt changes without running a regression suite first.

The core requirement is simple: when any prompt in the system changes, run the full eval suite against that prompt and surface a diff of the score changes. A change that improves aggregate performance while regressing on a specific input category is only visible if you have the category labels and the historical baseline to compare against. Without that infrastructure, regressions propagate silently into production.

Khattab et al. (2023) formalise this further in DSPy, which frames the entire prompt engineering pipeline as a compilation problem: prompts are not written by hand but optimised automatically against a metric, with the eval suite serving as the objective function the compiler is maximising.^#DSPy Even if you do not adopt DSPy's full automated optimisation approach, the core insight is correct: the eval suite is the specification, and prompt iteration is the search procedure. That framing makes the value of a rigorous eval dataset obvious — it is not test overhead, it is the target you are aiming at.

The prompt diff problem

One subtlety unique to prompt versioning: small wording changes can produce large and non-obvious output changes. Adding "be concise" to a system prompt may improve average output length but degrade faithfulness because the model omits qualifications. Removing "You are an expert in…" may slightly reduce confidence calibration in a way that takes fifty evaluations to detect statistically.

The implication is that prompt changes should be reviewed with the same scrutiny as code changes that touch critical paths — and the eval diff is the code review tool. A change that shows clean scores in aggregate but moves the distribution of individual scores (more 3s and 5s, fewer 4s) is not a neutral change; it is a change in reliability that the aggregate hides. Track score variance, not just mean score.

8. The ship decision

The ship decision is a judgment call, not a calculation — but it should be a judgment call informed by evidence. The eval infrastructure built in this lesson exists to produce that evidence. What follows is a practical framework for translating eval scores into a defensible shipping decision.

The most common shipping error is not shipping too early — it is shipping without having defined what "ready" means. A team that has not set a target score before the first eval run will always find a reason the current score is acceptable. A team with a pre-defined target has a clear, objective, repeatable criterion that cannot be adjusted after the results are in.

A second common error is treating the initial ship as the end of the eval process. Production is where the real input distribution lives. The queries that real users submit will contain patterns, phrasings, and intent varieties that your golden set did not anticipate. The eval infrastructure built before launch is the baseline; the production monitoring system is what tells you whether that baseline holds. They are one continuous process, not two separate phases.

References

1. #Validator Shankar, S., et al. (2024). Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences. Studies calibration of LLM-assisted evaluation against human preferences and reports systematic divergence when judge pipelines are insufficiently aligned. arXiv:2404.12272
2. #ROUGE Lin, C.-Y. (2004). ROUGE: A Package for Automatic Evaluation of Summaries. Introduces the ROUGE family of n-gram overlap metrics; remains the standard for summarisation evaluation. Proceedings of the ACL Workshop on Text Summarization Branches Out. aclanthology.org/W04-1013
3. #BERTScore Zhang, T., et al. (2019). BERTScore: Evaluating Text Generation with BERT. Uses contextual BERT embeddings to compute semantic similarity between generated and reference text; more robust to paraphrase than n-gram methods. ICLR 2020. arXiv:1904.09675
4. #Judge Zheng, L., et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. Establishes LLM-as-judge as a scalable evaluation methodology, reporting >80% agreement between GPT-4 judges and human expert annotators. NeurIPS 2023. arXiv:2306.05685
5. #Summarization Shen, T., et al. (2023). Large Language Models are Not Yet Human-Level Evaluators for Abstractive Summarization. Finds LLM evaluators inconsistent and dimension-dependent in abstractive summarisation evaluation, highlighting reliability limits for uncalibrated judge setups. arXiv:2305.13091
6. #RAGAS Es, S., et al. (2023). RAGAS: Automated Evaluation of Retrieval Augmented Generation. Introduces the four-metric RAGAS framework — context recall, context precision, faithfulness, and answer relevance — for component-level RAG evaluation without requiring ground-truth references. arXiv:2309.15217
7. #DSPy Khattab, O., et al. (2023). DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines. Frames prompt engineering as a compilation problem with eval metrics as the objective function; introduces automated prompt optimisation against a labelled eval set. arXiv:2310.03714
8. #Survey Guo, Q., et al. (2023). Evaluating Large Language Models: A Comprehensive Survey. Broad taxonomy of LLM evaluation dimensions including capability, alignment, and safety; covers automated and human evaluation methodologies across task types. arXiv:2310.19736
9. #Anthropic Anthropic. (2025). Define success criteria and build evaluations. Anthropic's documentation on constructing eval pipelines for Claude-based systems: evaluation criteria, dataset design, and test development workflows before deployment. platform.claude.com/docs/en/test-and-evaluate/develop-tests