RAG and Context Injection Chunking, Embeddings and Hybrid Search

Quick answer

Retrieval-augmented generation retrieves relevant information from an external source at inference time and injects it into the prompt, so the model reasons over evidence you provide instead of relying on what it memorised during training.

  • Retrieval quality is a hard ceiling on generation quality — the model cannot reason its way out of bad context.
  • Chunking is the most underestimated decision in the pipeline: it determines everything retrieval can ever find.
  • The counterintuitive fix for over-retrieval: retrieve more, inject less.

Every prompt you write is a bet on what the model already knows. For a large frontier model, that bet pays off on general questions — summarisation, classification, reasoning, code. It fails the moment you need the model to know something specific: the contents of your company's internal policy document, the state of a database record, the transcript of a call that happened this morning. No amount of prompt engineering retrieves information the model was never trained on.

Retrieval-augmented generation is the standard solution. Instead of relying on the model's parametric memory, you retrieve the relevant information from an external source at inference time and inject it into the prompt. The model's job shifts: it is no longer asked to know, but to reason over evidence you provide.

1. The knowledge problem

A language model's parametric knowledge has two hard limits: a training cutoff — the model does not know about events after a certain date — and a coverage limit — even within the training window, vast amounts of domain-specific, proprietary, or obscure information were never ingested.

The standard response is fine-tuning: adjust the model's weights on your domain data. This is often the wrong choice. It is expensive, slow, risks catastrophic forgetting, and conflates two different problems: teaching the model how to behave versus giving it what to know.

Fine-tuning adjusts the model's behaviour and style. RAG adjusts the model's knowledge at inference time. For dynamic, high-volume, or proprietary information — legal documents, product catalogues, support tickets, live data — RAG is almost always the right tool.

Lewis et al. (2020) introduced RAG as a formal architecture, combining a non-parametric retrieval module with a parametric sequence-to-sequence model, showing consistent improvements over closed-book generation across knowledge-intensive NLP benchmarks. The caveat is that "implemented well" carries more weight than it appears — most RAG failures are not failures of the underlying idea; they are failures of chunking, retrieval, or injection.

2. The RAG pipeline

A RAG system has three distinct stages, and understanding each independently is the prerequisite for diagnosing failures — a problem in stage one propagates silently through stages two and three, producing a confident, fluent, and entirely wrong answer.

Indexing

Documents are chunked, embedded, and stored in a searchable index — offline, before any query arrives.

Retrieval

A query is embedded and matched against the index to find the most relevant chunks.

Generation

Retrieved chunks are injected into the prompt and the model generates an answer grounded in them.

The pipeline's key property is that stages one and two are entirely separate from the language model — it only participates in stage three. You can swap the retrieval system, update the document index, or change the embedding model without touching your prompt or your generation model. It also means retrieval quality is a hard ceiling on generation quality. A model that receives poor or contradictory context cannot compensate with clever reasoning — it will either hallucinate to fill the gap or faithfully reflect the wrong context.

3. Chunking strategies

Chunking is the most underestimated decision in a RAG pipeline. It determines the granularity of what retrieval can find and controls the trade-off between precision and context. Chunk too small and you lose meaning; chunk too large and you pollute the context window with irrelevant content. There is no universally correct size — it depends on document type, embedding model, and query pattern.

Fixed-size chunking

Split the document into chunks of N tokens with an overlap of M tokens between adjacent chunks, so sentences split at a boundary still have their context available in at least one chunk. The simplest strategy, and the right starting point. 512 tokens suits most embedding models from 2022–2024; newer models supporting 8k+ token inputs may benefit from larger chunks.

512

tokens — fits most embedding models

64

tokens overlap — ~12%, preserves boundaries

4–6

chunks retrieved (top_k) to start

Start here. Tune based on retrieval eval, not intuition.

Semantic chunking

Split at natural semantic boundaries instead of fixed counts — paragraph breaks, section headings, or topic shifts detected by comparing embedding similarity between consecutive sentences. Produces more coherent chunks, which improves retrieval precision, at the cost of variable sizes that complicate context budgeting. Use it when document structure is irregular — scraped web content, mixed-format reports.

Hierarchical chunking

Index documents at multiple granularity levels simultaneously: whole sections for broad retrieval, paragraphs for precision. At query time, retrieve at the paragraph level but return the surrounding section as context — a "small-to-big" pattern especially effective for long technical documents where the relevant answer is one sentence that only makes sense in its section.

Four hierarchical variants

  • Parent document retriever. Index child chunks (paragraphs); when one matches, inject its parent section.
  • Summary indexing. Index LLM-generated summaries of each section; match against summaries, inject full sections.
  • Sentence window. Index individual sentences; when one matches, inject a window of ± N surrounding sentences.
  • Proposition chunking. Decompose documents into atomic, self-contained claims before embedding — maximum precision, high indexing cost.

Late chunking

A newer approach from Günther et al. (2024): embed the full document first, then chunk the resulting token embeddings rather than the raw text. Because the embeddings carry full document context, each chunk's vector captures the meaning of surrounding text — solving the "lost context" problem without hierarchical retrieval's complexity. Worth evaluating if your embedding model supports long contexts.

4. Embedding and similarity search

An embedding model maps text to a dense vector such that semantically similar texts land near each other by cosine distance — the foundation of semantic search: matching meaning instead of keywords.

General-purpose embedding models perform well on everyday language and degrade on domain-specific vocabulary — legal terms, medical nomenclature, proprietary product names — because those terms appeared infrequently in training data. The embedding model used at query time must be identical to the one used at index time. Mixing models produces nonsensical similarity scores; the vector spaces are not comparable, and retrieval silently fails.

For similarity search at scale, exact nearest-neighbour search across millions of vectors is computationally prohibitive. Approximate nearest-neighbour algorithms — most widely FAISS — partition the vector space using structures like HNSW or IVF, trading a small recall loss for sub-linear search. For production systems under roughly 100k documents, exact search is often fast enough and removes a source of failure.

Hybrid search

Semantic search excels at intent matching but fails on exact strings — product codes, names, rare terms never part of the embedding model's training distribution. Lexical search (BM25) excels at exact and rare-term matching but misses paraphrases and synonyms. Hybrid search runs both in parallel and fuses the result sets with Reciprocal Rank Fusion. Anthropic's guidance on contextual retrieval found that contextual chunk enrichment alone reduced retrieval failures by 49%, and by 67% combined with hybrid search and reranking.

Add a cross-encoder reranker as the final step — it scores query–chunk pairs jointly rather than separately, which is dramatically more accurate at the cost of latency. Apply it only to the top-20 semantic candidates, not the full index.

Search method → strength → weakness

  • Dense (semantic). Intent matching, paraphrase recall. Weak on rare terms and exact strings.
  • Sparse (BM25). Exact strings, rare tokens. Blind to synonyms.
  • Hybrid (RRF fusion). Both of the above, at the cost of more infrastructure — the production default.
  • Cross-encoder reranker. Highest precision, applied after hybrid retrieval to the top candidates only.

5. Context injection

Retrieval gives you relevant chunks. Injection is how you assemble them into a prompt the model can reason over correctly — a prompt engineering problem, not a retrieval problem, and where most RAG implementations leave quality on the table.

The naive approach — concatenate retrieved chunks as raw text and append them to the query — works, but tells the model nothing about what the text is, where it came from, or how confident the retrieval was. A model presented with a wall of undifferentiated text will hallucinate transitions between chunks and fail to attribute claims correctly.

The injection template

Structure your context injection. Each retrieved chunk should arrive with at minimum a source identifier, the chunk content, and a separator. The relevance label is derived from the retriever's similarity score — high, medium, or low based on thresholds you define; it is not automatically generated, your orchestration layer maps the float score to a label.

Structured injection template Example
SYSTEM: Answer using only the context below. Cite the
source for each claim. If the context does not contain
the answer, say so — do not guess.

CONTEXT:
--- Source: policy-handbook.pdf | Section: 4.2 | Relevance: high ---
[chunk content]

--- Source: faq-2026-q1.md | Section: n/a | Relevance: medium ---
[chunk content]

--- Source: support-ticket-8821.txt | Section: n/a | Relevance: low ---
[chunk content]

USER QUESTION: [user query]
Avoid — unstructured concatenation Common mistake
Here is some information: [chunk 1 text] [chunk 2 text]
[chunk 3 text]

Answer this question: [user query]

Problems: no source attribution, no boundary markers
between chunks, no instruction about how to handle missing
or conflicting information, no grounding constraint — the
model will freely mix retrieved content with parametric
knowledge, making hallucinations impossible to detect.

Contextual chunk enrichment

A chunk retrieved in isolation often loses its meaning — "the policy applies only to registered users" means nothing without knowing which policy, from which document. Contextual retrieval — prepending each chunk at index time with a brief LLM-generated summary of its document context — significantly improves accuracy because the embeddings now carry situational information, not just the raw sentence. It adds a one-time indexing cost for a dramatically improved runtime result — one of the highest-return interventions available in a standard RAG pipeline.

Before

"The policy applies only to registered users and excludes trial accounts from eligibility." — embedded in isolation; the retrieval model has no document context.

After

"This excerpt is from the Refund Policy section of the 2026 Customer Terms of Service document, which governs e-commerce transactions on the platform. The policy applies only to registered users and excludes trial accounts from eligibility." — embedded with prepended context; the retrieval model can match document-level intent.

6. The context window budget

Every token spent on retrieved context is a token unavailable for system instructions, conversation history, and the model's own reasoning. Managing this budget explicitly — rather than hoping it fits — is the difference between a system that degrades gracefully under load and one that silently truncates the most relevant chunk.

Frontier models now support 128k to 1M token windows, which makes overflow less likely, but two subtler problems remain. Liu et al. (2023) documented the "lost in the middle" effect: models attend less reliably to information placed in the middle of a long context than at the beginning or end. And injecting more context than necessary increases latency and cost without improving quality — sometimes degrading it, since the model must distinguish signal from noise across a larger set.

Working rule: reserve the beginning of the context window for the system prompt and hard constraints, place retrieved context immediately after, and place the user query last. Position your most important retrieved chunks first and last — attention is strongest at the boundaries of long contexts.

A 32k-token allocation, mid-complexity RAG

  • System prompt. 500–1,000 tokens — role, constraints, output format, grounding instruction.
  • Retrieved context. 8,000–16,000 tokens — top-k chunks, source metadata, separators.
  • Conversation history. 2,000–4,000 tokens — last N turns, summarised if needed.
  • User query. 200–500 tokens — the current message.
  • Generation headroom. 1,000–4,000 tokens — reserved for the model's output.

If retrieved content consistently exceeds the budget, the fix is not a larger top-k — it is better retrieval precision, so fewer chunks are needed to cover the relevant information. A retrieval system that must return ten chunks to be sure the right one is included needs to be fixed.

7. Failure modes

RAG systems fail differently from plain generation. A plain failure is visible — the model says it doesn't know. A RAG failure is often invisible: a confident, fluent answer grounded in the wrong content. Users trust it. The system logs a successful completion. The error propagates silently.

Four failure modes

  • Retrieval miss. The correct chunk exists but is not returned — vocabulary mismatch, a boundary cutting off the relevant sentence, or low embedding precision for the domain. Detect with recall@k against a labelled set; fix with hybrid search, reranking, or contextual enrichment.
  • Context poisoning. Retrieved chunks contradict each other or are outdated, and the model treats them as authoritative because the injection prompt says to. The most dangerous mode — the grounding instruction that prevents hallucination also prevents correcting bad context. Mitigate with document dates in metadata, explicit conflict-resolution rules, and post-retrieval deduplication.
  • Retrieval–answer mismatch. Retrieved chunks are topically related but do not contain the answer, so the model either falls back to parametric knowledge or hedges confidently. Detect with faithfulness scoring against the retrieved chunks.
  • Over-retrieval and context dilution. Too many injected chunks degrade quality even when the correct one is present. The fix is counterintuitive: retrieve more, inject less — a high top-k at retrieval, reranking, then only the top 4–6 injected.
Context poisoning — a conflicting-source example Failure
Your index contains two documents about the return window:

policy-2024.pdf: "Returns accepted within 14 days."
policy-2026.pdf: "Returns accepted within 30 days."

Retrieval returns both chunks. The model, instructed to
answer only from context, resolves the conflict
arbitrarily — often averaging or selecting the first
document. Neither answer is reliable.

Fix: add document date as metadata; instruct the model to
prefer the most recent source when content conflicts.

8. Decision guide

RAG is not the answer to every information access problem. Applying it indiscriminately adds infrastructure complexity that is unnecessary when the model's parametric knowledge is sufficient. Advanced architectures — Self-RAG (Asai et al., 2023), which teaches a model to decide when to retrieve and critique its own results, and FLARE (Jiang et al., 2023), which retrieves iteratively based on low-confidence predictions — address some of the failure modes above at the cost of complexity and latency. The right order is: build a working naive pipeline first, measure its failure modes against your real queries, then apply targeted improvements.

Use RAG

Information is proprietary, internal, or domain-specific. Information changes faster than a fine-tuning cycle allows. Answers must be auditable — traced to a specific source. The domain is narrow but deep (legal, medical, technical documentation). Query volume is high and information is frequently updated. RAG is well-suited: invest in indexing quality and retrieval evaluation.

Skip RAG

The information is stable, well-represented in the model's training data, and general in nature. The query requires reasoning, not retrieval. The document corpus is fewer than ~200,000 tokens total — include them in the system prompt directly using prompt caching to reduce cost. You need the model to behave differently, not know more. Fine-tuning, system prompt inclusion, or base model prompting may be sufficient.

Evaluate first

Build a labelled eval set of 50–100 question–answer pairs before committing to an architecture. Measure retrieval recall@k and answer faithfulness against the retrieved context. If retrieval recall is high and faithfulness is low, the problem is injection — not retrieval. If recall is low, fix chunking and embedding before anything else. Let the eval guide the architecture — most teams invert this order.

Lesson 06 closes the series with evaluation and testing: how to build the eval datasets, metrics, and automated pipelines needed to measure prompt and retrieval quality systematically — and how to know when a RAG system is actually ready to ship.

Was this lesson useful?

FAQ

Frequently asked questions

RAG grounds a language model's answers in external knowledge by retrieving relevant document chunks at query time and injecting them into the prompt. Introduced formally by Lewis et al. (2020), it adjusts what the model knows at inference time without retraining its weights, and has become the dominant architecture for grounding model output in verifiable, updatable information.

References

  1. Lewis, P., et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. NeurIPS 2020. arXiv:2005.11401.
  2. Gao, Y., et al. (2023). Retrieval-Augmented Generation for Large Language Models: A Survey. arXiv:2312.10997.
  3. Günther, M., et al. (2024). Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding Models. arXiv:2409.04701.
  4. Douze, M., et al. (2024). The Faiss Library. arXiv:2401.08281.
  5. Robertson, S. & Zaragoza, H. (2009). The Probabilistic Relevance Framework: BM25 and Beyond. Foundations and Trends in Information Retrieval, 3(4).
  6. Anthropic. (2024). Introducing Contextual Retrieval. anthropic.com/news/contextual-retrieval.
  7. Liu, N. F., et al. (2023). Lost in the Middle: How Language Models Use Long Contexts. arXiv:2307.03172.
  8. Asai, A., et al. (2023). Self-RAG: Learning to Retrieve, Generate, and Critique Through Self-Reflection. ICLR 2024. arXiv:2310.11511.
  9. Jiang, Z., et al. (2023). Active Retrieval Augmented Generation. arXiv:2305.06983.

Ready to test yourself?

10 questions drawn at random from the six lessons of Part I. No time limit. Share your score.

Take the Quiz

Contact

Got a system to build?

If you're building a RAG pipeline, an AI agent workflow, or an MCP integration and want to compare notes, swap ideas, or just avoid the mistakes I've already made — reach out.

Or get the Playbook in your inbox — one email when a new lesson ships, and nothing else.

No noise, no spam. Unsubscribe anytime.