Evaluation and Testing Eval Datasets, Judges and CI

Quick answer

Evaluation is what turns "this prompt is better" from an opinion into a reproducible claim. Because LLM outputs are stochastic, correctness is a property of the output distribution, not a single execution — evaluation measures that distribution instead of the examples you happened to check.

  • A mature framework has three layers: unit, integration, and system evals — each answering a different question.
  • A judge prompt is only as good as its rubric. "Rate 1 to 5" is not a rubric.
  • The most common shipping error is not shipping too early — it is shipping without defining what "ready" means.

Every earlier lesson in this series has involved a version of the same implicit claim: this prompt is better than that one. Better at what? Measured how? On which inputs? Compared to which baseline? If you cannot answer those questions with a number derived from a reproducible test, the claim is an opinion — and opinions about AI system quality have a remarkably poor track record in production.

Evaluation is the discipline that makes the claim rigorous. It is also the most consistently neglected part of the prompt engineering workflow. Teams iterate on prompts against three or four manually verified examples, ship to production, and discover the failure modes three months later in user complaints. Skipping it means you are not engineering; you are guessing at scale.

1. Why evaluation is the missing discipline

In traditional software, correctness is a property of an individual execution — a function's tests pass or they don't. Prompt engineering, practised without evals, has no equivalent: the prompt "feels right" or it does not, and that is not a foundation you can build a reliable system on.

The core difficulty is that language model outputs are stochastic and high-dimensional. The same prompt can produce different outputs on successive runs. In LLM systems, correctness is a property of the output distribution — not "does this return the right value?" but "does this system produce outputs in the acceptable quality band, reliably, across a representative distribution of inputs?" A prompt that produces excellent output on 80% of inputs and catastrophically wrong output on 20% is not a good prompt — it is a latent incident.

A secondary reason evaluation gets neglected: it requires making the quality bar explicit. "The output should be helpful and accurate" is not a specification — it is a wish. Writing an eval forces you to operationalise quality: accurate with respect to what ground truth, helpful by what rubric, evaluated against which input distribution. That exercise surfaces disagreements about what the system is supposed to do before those disagreements surface in production.

Shankar et al. (2024) demonstrate one dimension of this in their study of how teams calibrate LLM-assisted evaluation against human preferences — showing systematic divergence when judge pipelines are not aligned. In practice, teams without structured eval pipelines often overestimate system quality, misattribute failures to the wrong components, and regress on previously working capabilities when iterating on prompts without regression tests. These are recurring practitioner failure patterns to design against.

2. The evaluation stack

A mature evaluation framework has three layers, each answering a different question. Running only the top layer — end-to-end human evaluation — is expensive, slow, and gives no signal about where a failure originates. Running only the bottom layer tells you nothing about whether the full pipeline works. You need all three.

Unit evals

A single prompt against fixed input–output pairs. Cheap enough to run on every commit.

Integration evals

The full pipeline end to end. Tens to hundreds of API calls per run.

System evals

Human raters against a defined rubric — the ground truth that calibrates L1 and L2.

L1 and L2 run automatically on every prompt change — they are your CI pipeline. L3 runs periodically, weekly or before a major release, and its results calibrate the automated layers: if a judge prompt scores outputs highly that human raters consistently score poorly, the judge prompt needs revision. Design the pipeline so L1 catches most regressions before they reach L2, and L2 catches failures before they reach L3.

3. Building the eval dataset

An eval dataset is a collection of input, expected-output, and metadata triples against which your system is measured. A dataset that does not represent your true input distribution produces eval scores that are meaningless, or worse, misleading.

The golden set

Start with 50 to 100 manually verified examples — small enough to annotate carefully, large enough to detect meaningful differences between prompt versions, drawn from your actual or expected input distribution. A well-constructed golden set has four parts per example: the input, the ground-truth answer, a rubric defining a correct response, and a category label. The category label matters — an aggregate score of 82% hides whether the system fails uniformly or catastrophically on one subcategory.

Input

Should represent real query patterns, not just easy or obvious cases.

Reference

This is your quality ceiling. If the reference is wrong, your metric is measuring the wrong thing.

Rubric

Without a rubric, human raters disagree and automated judges are uncalibrated.

Category

E.g. factual lookup, multi-hop reasoning, ambiguous query, out-of-scope request, adversarial input. Enables slice analysis: which input types fail, and how often.

Adversarial and edge cases

A golden set drawn only from representative inputs will miss the failure modes that matter most. Dedicate at least 20% of the eval set to inputs designed to probe reliability: boundary inputs at the edge of the system's scope, ambiguous queries with multiple interpretations, conflicting context that contradicts the correct answer, and paraphrase variants of the same question. As the system matures, the highest-value additions are failure-derived cases — real production inputs where the system produced a wrong output. Every production failure should produce at least one new eval case.

What each category actually tests

  • Boundary inputs. Does the system correctly decline out-of-scope requests, or does it attempt an answer with no basis?
  • Ambiguous queries. Does the system ask for clarification or commit to one interpretation without acknowledging the ambiguity?
  • Conflicting context. Does the model faithfully reproduce the bad context, or does it resist?
  • Paraphrase variants. Significant quality variance across paraphrases reveals fragile prompts.

4. Metrics taxonomy

No single metric captures output quality across all tasks. Three tiers cover the range, from cheapest to most expressive.

Exact match and rule-based metrics

Exact string match, regex, and schema validation are the cheapest and most reliable metrics — when applicable. They apply when the output format is fully constrained: a specific JSON schema, a classification label from a fixed vocabulary, a boolean, a number. The failure mode is obvious: exact match is too strict for natural language generation. Use rule-based metrics for structure and factual keys, not for the content of generated text.

Semantic similarity

Reference-based metrics compare generated output against a reference using embeddings or n-gram overlap. ROUGE measures n-gram overlap and is the standard for summarisation. BERTScore computes token-level cosine similarity using contextual embeddings, more robust to paraphrase than n-gram methods. The limitation of all reference-based metrics: they require a reference, and penalise correct paraphrases that use different wording. High ROUGE does not mean high quality; low ROUGE does not mean low quality.

LLM-as-judge

The most expressive and increasingly dominant approach: a second, typically stronger, language model reads the input, the output, and optionally a reference, and scores it against a rubric. Zheng et al. (2023) formalise this in the MT-Bench framework, showing GPT-4 as judge achieves over 80% agreement with human expert evaluators on open-ended tasks — comparable to the agreement rate between two independent human annotators.

Metric → cost → best for

  • Exact match / rule-based. Near zero cost. Structured outputs, labels, schemas. Too strict for free-form text.
  • ROUGE / n-gram overlap. Near zero cost. Summarisation, translation. Penalises correct paraphrases.
  • BERTScore. Low cost (local model). Any task with a reference string. Requires a reference; misses factual errors.
  • LLM-as-judge. Medium cost (API call). Open-ended generation, RAG answers. Judge biases; requires calibration.
  • Human evaluation. High cost (annotator time). Ground truth and judge calibration. Slow, expensive, not scalable.

5. Designing the judge prompt

An LLM judge is only as good as its judge prompt. A vague one produces noisy, inconsistent scores that add variance without adding signal.

Four design principles

  • Score, don't compare. Asking the judge to pick the better of two outputs introduces position bias — it systematically favours whichever is presented first. Absolute scoring against a rubric is more consistent and easier to interpret.
  • Define every score level. "Rate 1 to 5" is not a rubric. Tell the judge specifically what differentiates a 3 from a 4, or scores cluster around the middle by default.
  • Reason before scoring. Forcing the judge to explain before committing to a score reduces positional anchoring and leaves a trace you can inspect — the chain-of-thought principle applied to evaluation.
  • Test for three known biases. Verbosity bias (longer scores higher regardless of quality — penalise unnecessary length explicitly); self-enhancement bias (a model judging its own output scores it too high — use a different model family); format bias (markdown and headers score higher than equal-quality plain prose — normalise formatting before scoring).
LLM-as-judge — faithfulness rubric System prompt
# Role: faithfulness judge.
You will read a set of source passages and a generated
response. Score how faithfully the response is grounded
in the passages, from 1 to 5.

1 — Response contains claims that directly contradict
    the passages.
2 — Response contains claims not supported by the
    passages, but not directly contradicting them.
3 — Response is mostly grounded, with one unsupported
    but plausible claim.
4 — Response is fully grounded, with minor imprecision
    in phrasing.
5 — All claims are directly traceable to specific
    sentences in the passages.

Reason step by step before scoring. Return JSON:
{"reasoning": "...", "score": 1-5, "issues": [...]}

Shen et al. (2023) found LLM evaluators inconsistent and dimension-dependent in their evaluation of abstractive summarisation — an early warning that uncalibrated judge pipelines produce unreliable signals. Calibrate your judge against human annotations before trusting it at scale. If judge and human raters disagree on more than 25% of examples, the fix is a better rubric, not a stronger model — rubric precision is almost always the bottleneck, not model capability.

6. Evaluating RAG pipelines

A RAG pipeline has two components that must be evaluated independently: retrieval and generation. Evaluating only the final output makes it impossible to localise failures — a wrong answer could mean the right chunks were retrieved but ignored, or the wrong chunks were retrieved and faithfully reproduced — the eight irrelevant chunks are diluting the signal and potentially triggering the over-retrieval failures described in Lesson 05. Es et al. (2023) introduce RAGAS, a largely reference-free framework that decomposes RAG quality into four independent metrics.

Four RAGAS metrics

  • Context recall. Do the retrieved chunks contain enough to answer the question? Reference needed: yes. Low recall points at chunking or retrieval configuration, not the generation prompt.
  • Context precision. What proportion of retrieved chunks are actually relevant? Reference needed: no — computable with a judge scoring each chunk's relevance independently.
  • Faithfulness. Is every claim in the answer supported by the retrieved context? Reference needed: no. Not whether the answer is correct — whether it is grounded. A faithful answer built on wrong context is still a faithfulness success; that is a retrieval problem, not a generation one.
  • Answer relevance. Does the answer actually address the question asked? Reference needed: no. Computed by asking a judge whether the question is fully answered, partially answered, or evaded — and it catches this hedging pattern that faithfulness scores miss entirely.

Run all four on every eval set update and track them as separate time series. A prompt change that improves faithfulness while degrading context precision is telling you something specific about what changed — and that specificity is what makes iteration tractable.

7. Regression testing and CI

A prompt is a software artifact. It should be version-controlled, reviewed, and tested before deployment, the same way any other code change is. The core requirement: when any prompt changes, run the full eval suite against it and surface a diff of the score changes. A change that improves aggregate performance while regressing on a specific category is only visible with category labels and a historical baseline to compare against. Without that infrastructure, regressions propagate silently into production.

Khattab et al. (2023) frame the entire prompt engineering pipeline as a compilation problem in DSPy: prompts are optimised automatically against a metric, with the eval suite as the objective function. Even without adopting automated optimisation, the insight holds — the eval suite is the specification, and prompt iteration is the search procedure.

Automated eval pipeline — CI integration Pipeline
1. Load eval dataset (golden set + adversarial cases)
2. Run system on all inputs
3. Score outputs (rule-based + judge, as applicable)
4. Aggregate by category
5. Diff against baseline
6. Post report to PR

# Gate merge on: no category regression > threshold (e.g. 3%).
# Human review required for: any adversarial category change.

The prompt diff problem

Small wording changes can produce large, non-obvious output changes. Adding "be concise" may improve average length but degrade faithfulness because the model omits qualifications. Removing "You are an expert in…" may slightly reduce confidence calibration in a way that takes fifty evaluations to detect statistically. Prompt changes deserve the same scrutiny as code touching critical paths — the eval diff is the code review tool. A change that shows clean scores in aggregate but moves the distribution of individual scores (more 3s and 5s, fewer 4s) is not a neutral change; it is a change in reliability the aggregate hides. Track score variance, not just the mean.

8. The ship decision

The ship decision is a judgment call, not a calculation — but it should be a judgment call informed by evidence. The most common shipping error is not shipping too early; it is shipping without having defined what "ready" means. A team without a pre-defined target score will always find a reason the current score is acceptable.

A second common error: treating the initial ship as the end of the eval process. Production is where the real input distribution lives — queries real users submit contain patterns your golden set did not anticipate. The eval infrastructure built before launch is the baseline; production monitoring is what tells you whether that baseline holds. They are one continuous process, not two separate phases.

Required

Hard blockers that must pass before shipping. Typically: no regression on adversarial and safety-critical eval categories, faithfulness score above a minimum threshold on all input types, structured output format compliance at 100% on the golden set. Non-negotiable — a system that fails a hard blocker does not ship.

Target

The quality bar you are aiming for on primary metrics (faithfulness, answer relevance, aggregate judge score). Defined before the first eval run, not after — setting targets retrospectively is confirmation bias by another name. If the system meets the target, it is a ship. If not, the gap is the work.

Known gaps

Input categories or edge cases where the system underperforms the target but is not a hard blocker. Acknowledged, logged, and prioritised for the next iteration. Every system ships with known gaps — the question is whether the gaps are acceptable for the use case.

Monitoring

A plan for what you will measure in production: which outputs will be sampled, who will review them, and what triggers a rollback. Eval in staging and eval in production are both necessary. Neither replaces the other.

Lesson 01 established that a prompt shapes a probability distribution. Every lesson since has been about methods for moving that distribution toward the region of useful, accurate outputs. Evaluation is how you measure whether you succeeded — not on the examples you designed for, but on the full distribution of inputs your system will actually encounter.

Without evaluation, every claim about quality is a prior, not a posterior. It is where prompt engineering becomes a discipline rather than a craft.
Closing the loop across the series

Was this lesson useful?

FAQ

Frequently asked questions

Language model outputs are stochastic and high-dimensional: the same prompt produces different outputs across runs, so correctness is a property of the output distribution rather than a single execution. A prompt that is excellent on 80% of inputs and catastrophically wrong on 20% is a latent incident — evaluation measures the full distribution instead of the examples you happened to check.

References

  1. Shankar, S., et al. (2024). Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences. arXiv:2404.12272.
  2. Lin, C.-Y. (2004). ROUGE: A Package for Automatic Evaluation of Summaries. ACL Workshop on Text Summarization Branches Out.
  3. Zhang, T., et al. (2019). BERTScore: Evaluating Text Generation with BERT. ICLR 2020. arXiv:1904.09675.
  4. Zheng, L., et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS 2023. arXiv:2306.05685.
  5. Shen, T., et al. (2023). Large Language Models are Not Yet Human-Level Evaluators for Abstractive Summarization. arXiv:2305.13091.
  6. Es, S., et al. (2023). RAGAS: Automated Evaluation of Retrieval Augmented Generation. arXiv:2309.15217.
  7. Khattab, O., et al. (2023). DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines. arXiv:2310.03714.
  8. Guo, Q., et al. (2023). Evaluating Large Language Models: A Comprehensive Survey. arXiv:2310.19736.
  9. Anthropic. (2025). Define success criteria and build evaluations. platform.claude.com/docs.

Ready to test yourself?

10 questions drawn at random from the six lessons of Part I. No time limit. Share your score.

Take the Quiz

Contact

Got a system to build?

If you're building a RAG pipeline, an AI agent workflow, or an MCP integration and want to compare notes, swap ideas, or just avoid the mistakes I've already made — reach out.

Or get the Playbook in your inbox — one email when a new lesson ships, and nothing else.

No noise, no spam. Unsubscribe anytime.