Every prompt you write is a bet on what the model already knows. For a large frontier model trained on a broad corpus, that bet pays off on general questions — summarisation, classification, reasoning, code. It fails the moment you need the model to know something specific: the contents of your company's internal policy document, the state of a database record, the transcript of a call that happened this morning. No amount of prompt engineering retrieves information the model was never trained on.
Retrieval-augmented generation is the standard solution. Instead of relying on the model's parametric memory, you retrieve the relevant information from an external source at inference time and inject it into the prompt. The model's job shifts: it is no longer asked to know, but to reason over evidence you provide. That shift, and precisely how you execute it, is what this lesson is about.
1. The knowledge problem
A language model's parametric knowledge — the information encoded into its weights during training — has two hard limits. The first is a training cutoff: the model does not know about events after a certain date. The second is a coverage limit: even within the training window, vast amounts of domain-specific, proprietary, or simply obscure information were never ingested.
The standard response is fine-tuning: adjust the model's weights on your domain data so the information becomes part of its parametric memory. This is often the wrong choice. Fine-tuning is expensive, slow, and introduces its own failure modes — notably the risk of catastrophic forgetting — performance loss on general tasks — particularly with full fine-tuning on small or narrow datasets. More importantly, it conflates two very different problems: teaching the model how to behave versus giving it what to know.
The core distinction
Fine-tuning adjusts the model's behaviour and style. RAG adjusts the model's knowledge at inference time. For dynamic, high-volume, or proprietary information — legal documents, product catalogues, support tickets, live data — RAG is almost always the right tool. Fine-tuning is for cases where the model needs to reason and respond differently, not just know more.
Lewis et al. (2020) introduced RAG as a formal architecture, combining a non-parametric retrieval module with a parametric sequence-to-sequence model, showing consistent improvements over closed-book generation across knowledge-intensive NLP benchmarks.#NeurIPS Since then, RAG has become the dominant architecture for any system that needs to ground model output in verifiable, updatable, domain-specific information.
The practical implication is clear: before you reach for fine-tuning, you should ask whether the problem is one of knowledge access rather than behavioural calibration. If the answer is yes, RAG — implemented well — will be faster to build, cheaper to run, and easier to audit. The caveat is that "implemented well" carries more weight than it appears. Most RAG failures are not failures of the underlying idea; they are failures of chunking, retrieval, or injection.
2. The RAG pipeline
A RAG system has three distinct stages. Understanding each stage independently is the prerequisite for diagnosing failures, because a problem in stage one propagates silently through stages two and three — producing a confident, fluent, and entirely wrong answer.
The pipeline's key property is that stages one and two are entirely separate from the language model. The LLM only participates in stage three. This means you can swap the retrieval system, update the document index, or change the embedding model without touching your prompt or your generation model. The pipeline is modular by design.
It also means that retrieval quality is a hard ceiling on generation quality. A model that receives poor, irrelevant, or contradictory context cannot compensate with clever reasoning — it will either hallucinate to fill the gap or produce an answer that faithfully reflects the wrong context. Garbage in, garbage out applies at every stage, but the retrieval stage compounds the error before the LLM has any opportunity to correct it.
3. Chunking strategies
Chunking is the most underestimated decision in a RAG pipeline. It determines the granularity of what the retrieval system can find, and it controls the trade-off between precision and context. Chunk too small and you lose meaning; chunk too large and you pollute the context window with irrelevant content that dilutes the signal.
There is no universally correct chunk size. The right choice depends on your document type, your embedding model's token limit, and the nature of the queries your system will receive. What follows are the four main strategies and when each applies.
Fixed-size chunking
Split the document into chunks of N tokens (or characters) with an overlap of M tokens between adjacent chunks. The overlap ensures that sentences split at a chunk boundary still have their context available in at least one chunk. This is the simplest strategy and the right starting point.
Note: 512 tokens suits most embedding models from 2022–2024. Newer models supporting 8k+ token inputs may benefit from larger chunk sizes — consult your embedding model's documentation for its optimal input length.
Semantic chunking
Instead of splitting at fixed token counts, split at natural semantic boundaries — paragraph breaks, section headings, or topic shifts detected by comparing embedding similarity between consecutive sentences. When the cosine similarity between sentence n and sentence n+1 drops below a threshold, you have found a boundary.
Semantic chunking produces chunks that are more coherent — each chunk is more likely to discuss one thing — which improves retrieval precision. The trade-off is variable chunk sizes, which complicate context window budgeting downstream. Use this strategy when document structure is irregular (e.g., scraped web content, mixed-format reports) and fixed splitting would routinely break logical units apart.
Hierarchical chunking
Index documents at multiple granularity levels simultaneously: whole sections for broad thematic retrieval, paragraphs for precision retrieval. At query time, retrieve at the paragraph level but return the surrounding section as context. This "small-to-big" pattern — retrieve small chunks, inject large ones — is particularly effective for long technical documents where the relevant answer is a specific sentence, but that sentence only makes sense in its section context.#Survey
Parent document retriever
Index child chunks (paragraphs). When a child matches, inject its parent (section) as context. Retrieval precision is high; injected context is rich.
Summary indexing
Index LLM-generated summaries of each section. Match against summaries; inject full sections. Best when documents are dense and query intent is thematic.
Sentence window
Index individual sentences. When a sentence matches, inject a window of ±N surrounding sentences. High precision, controllable context size.
Proposition chunking
Decompose documents into atomic, self-contained propositions before embedding. Each chunk is a single verifiable claim. Maximum retrieval precision; high indexing cost.
Late chunking
A newer approach from Günther et al. (2024): embed the full document first, then chunk the resulting token embeddings rather than the raw text.#Late Because the embeddings are computed with full document context, each chunk's vector representation captures the meaning of the surrounding text — solving the "lost context" problem inherent to fixed-size chunking without the complexity of hierarchical retrieval. If your embedding model supports long contexts (>8k tokens), this is worth evaluating.
4. Embedding and similarity search
An embedding model maps a piece of text to a dense vector in a high-dimensional space, such that semantically similar texts land near each other by cosine distance. This is the foundation of semantic search: instead of matching keywords, you match meaning.
The choice of embedding model matters more than most teams expect, and it matters differently
depending on your domain. General-purpose embedding models (e.g., OpenAI's
text-embedding-3-large, Cohere's embed-v3, or open-source alternatives
like bge-m3) perform well on everyday language. They degrade on domain-specific
vocabulary — legal terms, medical nomenclature, proprietary product names — because those terms
appeared infrequently in training data and their embeddings are imprecise.
Critical rule
The embedding model used at query time must be identical to the one used at index time. If you re-embed your documents with a new model, you must rebuild the entire index. Mixing embedding models produces nonsensical similarity scores — the vector spaces are not comparable, and retrieval will silently fail.
For similarity search at scale, exact nearest-neighbour search across millions of vectors is computationally prohibitive. Approximate nearest-neighbour (ANN) algorithms — the most widely used being FAISS (Meta AI Similarity Search) — partition the vector space using structures like HNSW (Hierarchical Navigable Small World graphs) or IVF (Inverted File Index), enabling sub-linear search at the cost of a small recall trade-off.#FAISS
For production systems with fewer than ~100k documents, exact search is often fast enough and removes a source of failure. Use ANN indexing when you need it, not by default.
Hybrid search
Semantic search excels at intent matching but fails on exact strings: product codes, names, identifiers, and rare terms that were never part of the embedding model's training distribution. Lexical search (BM25 or similar) excels at exact and rare-term matching but misses paraphrases and synonyms.#BM25
Hybrid search combines both signals. In practice: run semantic search and BM25 in parallel, then fuse the result sets using Reciprocal Rank Fusion (RRF) or a learned linear combination. Anthropic's own guidance on contextual retrieval found that contextual chunk enrichment alone reduced retrieval failures by 49%, and when combined with hybrid search and reranking, by 67%.#Anthropic
Add a cross-encoder reranker as the final step. Cross-encoders score query–chunk pairs jointly (rather than separately, as bi-encoder embedding models do) and are dramatically more accurate — at the cost of latency. Apply reranking only to the top-20 semantic candidates, not to the full index.
| Search method | Strength | Weakness | When to use |
|---|---|---|---|
| Dense (semantic) | Intent matching, paraphrase recall | Rare terms, exact strings | Most conversational queries |
| Sparse (BM25) | Exact strings, rare tokens | Synonym blindness | IDs, codes, proper nouns |
| Hybrid (RRF fusion) | Both of the above | More infrastructure | Production default |
| Cross-encoder reranker | Highest precision | Latency; only top-k | After hybrid retrieval |
5. Context injection
Retrieval gives you relevant chunks. Injection is the question of how to assemble those chunks into a prompt that the model can reason over correctly. It is a prompt engineering problem, not a retrieval problem — and it is where most RAG implementations leave significant quality on the table.
The naive approach: concatenate retrieved chunks as raw text and append them to the user query. This works, but it fails to tell the model what the text is, where it came from, how confident the retrieval was, or how to handle conflicts between chunks. A model presented with a wall of undifferentiated text will do its best, but "doing its best" includes hallucinating transitions between chunks and failing to attribute claims correctly.
The injection template
Structure your context injection. Each retrieved chunk should arrive in the prompt with at minimum: a source identifier, the chunk content, and a separator. A well-designed injection template looks like this:
The relevance label shown here is derived from the retriever's similarity score — high, medium, or low based on thresholds you define. It is not automatically generated; your orchestration layer must map the float score to a label and insert it into the template.
Contextual chunk enrichment
A chunk retrieved in isolation often loses its meaning. The sentence "The policy applies only to registered users" means nothing without knowing which policy, in what context, from which document. Contextual retrieval — prepending each chunk at index time with a brief LLM-generated summary of its document context — significantly improves retrieval accuracy because the embeddings now carry situational information, not just the raw sentence.#Anthropic
This enrichment step adds a one-time LLM cost at index build time — generating a short contextual prefix for each chunk — but it is a fixed cost that dramatically improves runtime retrieval quality. It is one of the highest-return interventions available in a standard RAG pipeline.
6. The context window budget
A context window has a fixed capacity, and every token you spend on retrieved context is a token unavailable for system instructions, conversation history, and the model's own chain-of-thought reasoning. Managing this budget explicitly — rather than hoping it fits — is the difference between a system that degrades gracefully under load and one that silently truncates the most relevant chunk.
In practice, frontier models now support context windows of 128k to 1M tokens, which makes overflow less likely. But two subtler problems remain. First, very long contexts cause models to attend less reliably to information in the middle — the "lost in the middle" effect documented by Liu et al. (2023): model performance degrades significantly when relevant information appears in the middle of a long context, as opposed to the beginning or end.#Middle Second, injecting more context than necessary increases latency and cost without improving answer quality — and sometimes degrades it, because the model must now distinguish signal from noise across a larger document set.
Context budget allocation — working rule
Reserve the beginning of the context window for the system prompt and hard constraints. Place retrieved context immediately after. Place the user query last, immediately before the expected response. Position your most important retrieved chunks first and last — the model's attention is strongest at the boundaries of long contexts.
A practical allocation for a mid-complexity RAG system over a 32k token context:
| Context region | Token budget | Contents |
|---|---|---|
| System prompt | 500–1,000 | Role, constraints, output format, grounding instruction |
| Retrieved context | 8,000–16,000 | Top-k chunks, source metadata, separator markers |
| Conversation history | 2,000–4,000 | Last N turns, summarised if needed |
| User query | 200–500 | Current user message |
| Generation headroom | 1,000–4,000 | Reserved for model output (max_tokens) |
If retrieved content consistently exceeds the budget, the correct response is not to increase top-k or chunk size — it is to improve retrieval precision so you need fewer chunks to cover the relevant information. A retrieval system that must return 10 chunks to be sure the right one is included is a retrieval system that needs to be fixed.
7. Failure modes
RAG systems fail in ways that are qualitatively different from plain generation failures. A plain generation failure is visible — the model says "I don't know" or produces something obviously wrong. A RAG failure is often invisible: the model produces a confident, well-structured, fluent answer that is grounded in the wrong content. Users trust it. The system logs a successful completion. The error propagates silently.
Understanding the four principal failure modes is the prerequisite for building evaluation pipelines that actually catch them.
Retrieval miss
The correct chunk exists in the index but is not returned. Causes include: the query is phrased differently from the document (a vocabulary mismatch that semantic search partially mitigates but does not eliminate), the chunk boundary cuts off the relevant sentence, or the embedding model has low precision for this domain. Detection: measure recall@k against a labelled query–answer dataset. Fix: hybrid search, reranking, query expansion, or contextual chunk enrichment.
Context poisoning
Retrieved chunks contain contradictory, outdated, or factually incorrect content — and the model treats it as authoritative because the injection prompt instructs it to rely on the provided context. This is arguably the most dangerous failure mode because it is architecturally induced: the grounding instruction that prevents hallucination also prevents the model from correcting bad context using its parametric knowledge.
Mitigations: index metadata including document date and version; instruct the model explicitly on conflict resolution rules ("when sources contradict, prefer the most recent dated source"); implement a post-retrieval deduplication step that removes superseded document versions.
Retrieval–answer mismatch
The retrieved chunks are topically related to the query but do not actually contain the information needed to answer it. The model, instructed to answer from context, either generates an answer from its parametric knowledge while appearing to cite the retrieved chunks, or produces a confident hedged answer ("Based on the available information, it appears…") that gives no indication of the underlying retrieval failure.
This failure is common when chunk size is too small (the relevant sentence is present in the corpus but split across two chunks, neither of which passes the retrieval threshold on its own) or when the query requires synthesising information across multiple documents that are not retrieved together. Detection: faithfulness scoring — compare the generated answer against the retrieved chunks to verify that every claim in the answer is supported by the context provided.
Over-retrieval and context dilution
Injecting too many chunks degrades answer quality even when the correct chunk is present. The model must attend to more content, relevant signal is diluted by irrelevant context, and the probability of the model attending to the correct chunk decreases. Liu et al. (2023) showed that model performance on multi-document QA degrades non-linearly as the number of injected documents increases, with the steepest drops when relevant information is positioned in the middle of the context.#Middle
The fix is counterintuitive: retrieve more, inject less. Use a high top-k in the retrieval stage (e.g., 20 candidates), apply reranking, and inject only the top 4–6 after reranking. Precision at injection time matters more than recall at retrieval time.
8. Decision guide
RAG is not the answer to every information access problem. Applying it indiscriminately adds infrastructure complexity and failure modes that are unnecessary when the model's parametric knowledge is sufficient. Use the following framework before reaching for a vector database.
Advanced RAG architectures — Self-RAG (Asai et al., 2023), which teaches the model to decide when to retrieve and to critique its own retrieved content#Self-RAG; FLARE (Jiang et al., 2023), which retrieves iteratively based on low-confidence token predictions#FLARE — address some of the failure modes described above, at the cost of additional complexity and latency. They are worth knowing, but the right order is: build a working naive RAG pipeline first, measure its failure modes against your specific queries, and then apply targeted improvements. Complexity added before evaluation is complexity you cannot justify.
What comes next
Lesson 06 closes the series with evaluation and testing: how to build the eval datasets, metrics, and automated pipelines needed to measure prompt and retrieval quality systematically — and how to know when a RAG system is actually ready to ship.
Ready to test yourself?
10 questions on prompt engineering fundamentals. No time limit. Share your score with your network.
Take the Quiz →Frequently Asked Questions
What is retrieval-augmented generation (RAG)?
RAG grounds a language model's answers in external knowledge by retrieving relevant document chunks at query time and injecting them into the prompt. Introduced formally by Lewis et al. (2020), it adjusts what the model knows at inference time — without retraining its weights — and has become the dominant architecture for grounding model output in verifiable, updatable information.
When should you use RAG instead of fine-tuning?
Fine-tuning adjusts behaviour and style; RAG adjusts knowledge at inference time. For dynamic, high-volume, or proprietary information — legal documents, product catalogues, support tickets, live data — RAG is almost always the right tool. Reach for fine-tuning when the model needs to reason or respond differently, not just know more.
Why is retrieval quality a hard ceiling on RAG performance?
The language model only participates in the final stage of the pipeline. If retrieval returns poor, irrelevant, or contradictory context, the model cannot compensate with clever reasoning — it either hallucinates to fill the gap or faithfully reflects the wrong context. Errors in indexing and retrieval compound before the model has any opportunity to correct them.
Which chunking strategy should you use?
Fixed-size chunking is the fast baseline; semantic chunking splits on meaning boundaries; hierarchical chunking preserves document structure; late chunking embeds long context before splitting so each chunk retains global information. The right choice depends on document structure and query patterns — chunking decides what can ever be retrieved, so it deserves evaluation rather than defaults.
When should you skip RAG entirely?
When the information is stable, general, and well-represented in training data; when the query requires reasoning rather than retrieval; when the corpus is under roughly 200,000 tokens (include it directly in the system prompt with prompt caching); or when the real problem is behaviour, not knowledge. Build a labelled eval set of 50–100 question–answer pairs first, and let retrieval recall and answer faithfulness guide the architecture.
References
- #NeurIPS Lewis, P., et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. The foundational RAG paper, combining a DPR retriever with a BART generator; demonstrated consistent improvements over closed-book generation on open-domain QA. NeurIPS 2020. arXiv:2005.11401
- #Survey Gao, Y., et al. (2023). Retrieval-Augmented Generation for Large Language Models: A Survey. Comprehensive taxonomy of RAG architectures including naive, advanced, and modular RAG; covers chunking, indexing, retrieval, and generation strategies. arXiv:2312.10997
- #Late Günther, M., et al. (2024). Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding Models. Proposes embedding the full document before chunking to preserve cross-chunk context in each chunk's vector representation. arXiv:2409.04701
- #FAISS Douze, M., et al. (2024). The Faiss Library. Technical overview of FAISS, covering IVF, HNSW, and product quantisation indexing strategies for billion-scale approximate nearest-neighbour search. arXiv:2401.08281
- #BM25 Robertson, S. & Zaragoza, H. (2009). The Probabilistic Relevance Framework: BM25 and Beyond. Foundations of the BM25 ranking function. Remains the dominant sparse retrieval baseline in hybrid search systems. Foundations and Trends in Information Retrieval, 3(4). doi:10.1561/1500000019
- #Anthropic Anthropic. (2024). Introducing Contextual Retrieval. Describes the contextual chunk enrichment technique — prepending LLM-generated document context to each chunk at index time — reporting 67% reduction in retrieval failures when combined with hybrid search and reranking. anthropic.com/news/contextual-retrieval
- #Middle Liu, N. F., et al. (2023). Lost in the Middle: How Language Models Use Long Contexts. Demonstrates that LLM performance on multi-document QA degrades when relevant information is positioned in the middle of long contexts; informs chunk ordering strategy during injection. arXiv:2307.03172
- #Self-RAG Asai, A., et al. (2023). Self-RAG: Learning to Retrieve, Generate, and Critique Through Self-Reflection. Trains the model to generate special reflection tokens to decide when to retrieve and to score the relevance and support of retrieved passages. ICLR 2024. arXiv:2310.11511
- #FLARE Jiang, Z., et al. (2023). Active Retrieval Augmented Generation. Introduces FLARE: the model generates a tentative continuation, identifies low-confidence tokens as retrieval triggers, and retrieves relevant passages to refine the output. arXiv:2305.06983