Quick answer
Prompt chaining breaks a complex task into a sequence of simpler sub-tasks, each handled by a dedicated prompt, where one step's output becomes the next step's input. The model never sees the whole problem — only the sub-problem it is currently responsible for.
- Four architectures cover most real pipelines: sequential, branching, looped, map-reduce.
- A chain is only as reliable as its weakest boundary — validate every step's output before the next step consumes it.
- Chaining is not free. Use the minimum viable chain, not the maximum imaginable one.
You've written a good system prompt. You've added chain-of-thought. You're getting reasonable outputs on simple tasks. Then a real requirement lands: analyse a forty-page document, identify the three most critical risks, cross-reference them against a compliance checklist, draft a remediation summary, and format everything as a structured report. You try it in a single prompt. The model produces something that looks plausible — and is wrong in ways that are easy to miss.
This is not a model capability problem. It is a task decomposition problem. Prompt chaining is the principled answer to it.
1. What is prompt chaining?
Prompt chaining is the practice of breaking a complex task into a sequence of simpler sub-tasks, each handled by a dedicated prompt, where the output of one step becomes part of the input of the next. Wu et al. (2022) define a prompt chain precisely this way, and their AI Chains framing made these sequences transparent and controllable — an accurate mental model that foreshadowed today's DAG-based agent frameworks like LangGraph.
A model that does one thing reliably is more valuable than a model that tries to do ten things at once. Chaining is not about the model's ability — it is about constraining each call to a scope where the model can be excellent rather than merely adequate.
Dohan et al. (2022) formalise this further under the term language model cascades — probabilistic programs where each step conditions on the outputs of previous steps and each transition can incorporate deterministic logic. A chain is not just a sequence of LLM calls; it is a program in which LLM calls are the operations and string-formatted state is the memory.
Input
The output of the previous step — or the narrowest slice of it the step actually needs. Never the full upstream history unless it is truly required.
System prompt
A tight, single-responsibility instruction for this step alone. No multi-role prompts. One job per step.
Output
A structured, predictable artifact the next step can consume without re-interpretation. Define the contract before writing the prompt.
Validation
An optional assertion layer that confirms the output matches its schema before it is passed downstream. Cheap insurance against error propagation.
2. Why monolithic prompts break at scale
Before reaching for a chain, it is worth understanding precisely why a single prompt fails on complex tasks. The failure is not random — it follows predictable patterns.
Attention diffusion
Transformer attention is distributed across all tokens in the context window. When a prompt contains a task description, a long document, several examples, formatting rules, and output constraints simultaneously, the model's effective attention on any individual component decreases — instructions near the middle of a long prompt are consistently under-weighted relative to those at the beginning and end. A chain localises each step's attention to only the tokens that matter for that step.
Format contract conflicts
Complex tasks often require the model to simultaneously reason, retrieve, classify, and format. A model asked to analyse a contract, identify liability clauses, check each against a rubric, and output a JSON report with a supporting narrative is being asked to write in three registers — analytical prose, structured data, evaluative commentary — inside a single generation pass. Reliability degrades because the model cannot fully commit to any single format.
Error compounding
In a monolithic prompt, an incorrect inference in the first half of the response tends to propagate: because the model generates autoregressively, a flawed claim becomes context for everything that follows. In a chain, errors are local to the step in which they occur and can be caught at the boundary before they contaminate downstream steps.
Each prompt has a reliability ceiling — a complexity level above which output quality becomes unpredictable. Chaining does not raise the ceiling; it ensures you never hit it.
3. The four chain architectures
Four architectures cover the majority of real-world chaining needs. They are not mutually exclusive — production pipelines frequently combine all four — but it is useful to understand each in isolation first.
Sequential
Steps run in a fixed order, each step's output feeding the next. The simplest, most common pattern, and the right default for any task expressible as a series of transformations.
Branching
A classification step gates which sub-chain runs next. Correct when the same input type needs qualitatively different handling depending on content.
Looped / iterative
A step's output feeds back in until a termination condition is met — the canonical generate-critique-revise cycle.
Map-reduce
A large input is split into chunks processed independently, then merged by a final aggregation step.
Sequential chains
A practical example: producing a competitive analysis from raw research notes. The pipeline might extract structured facts from each source, normalise entities across sources, identify contradictions, rank findings by relevance, then draft the analysis. Each step is small, verifiable, and independently debuggable.
# Role: structured-extraction engine.
Extract every explicit claim from the source text below.
Return a JSON array of objects with fields: claim, subject,
source_sentence.
Rules:
- Extract only explicit claims. Do not infer.
- If the text contains no extractable claims, return
an empty array: []
- Do not add commentary, preamble, or explanation
outside the JSON.
Branching chains
The critical requirement is that the classification step produces a finite, deterministic set of labels that map cleanly to downstream chains. If the label space is fuzzy, the branch logic becomes unreliable — constrain the classifier to a fixed vocabulary and use structured output to enforce it.
Looped / iterative chains
Madaan et al. (2023) formalise this as Self-Refine — a model iteratively improves its own output using feedback generated by itself. A model acting as critic and reviser in separate, dedicated prompts outperforms a model asked to produce a final answer in a single pass, even at constant total token budget.
# Role: technical-editor critic. Review the draft against this rubric: 1. Every claim is traceable to the source material. 2. No sentence exceeds 30 words. 3. Section headings match the outline. Return JSON: {"pass": bool, "score": 1-10, "issues": [...], "revised_draft": "..."} # Orchestrator logic (not part of the prompt): # if response.pass === true -> exit loop # MAX_ITER=3 is a common starting point; tune per task.
Define the termination condition before you write the loop: a pass/fail gate, a fixed iteration cap, or a diminishing-returns threshold. In production, always combine a quality gate with a hard cap — an infinite loop is a live incident waiting to happen.
Map-reduce chains
The reduce step is the hardest part to get right. A naive "summarise all summaries" reduce prompt produces flattened, generic output. A better approach has the reduce step perform a specific aggregation operation — ranking, contradiction detection, theme extraction — rather than free-form merging.
Architecture → best for → key risk
- Sequential. Fixed transformation pipelines, document processing. Risk: error propagation with no boundary validation. Terminates: fixed — last step completes.
- Branching. Query routing, content triage, multi-domain tasks. Risk: label ambiguity causing misrouting. Terminates: fixed — branch terminates normally.
- Looped. Drafting, code generation, quality-gated output. Risk: infinite loops; diminishing returns after two or three passes. Terminates: conditional — gate or cap.
- Map-reduce. Long documents, batch summarisation, parallel extraction. Risk: the reduce step loses nuance; chunk-boundary artifacts. Terminates: fixed — reduce completes.
4. The router and dispatcher pattern
The router pattern deserves its own section because it appears at the entry point of virtually every non-trivial multi-chain system, and its quality determines the reliability of everything downstream. A router is a prompt (or a lightweight function wrapping one) that takes an incoming input and decides which chain should handle it.
The temptation is to write a router that tries to understand the input deeply. Resist it. A router's only job is classification into a finite set of categories — mutually exclusive, as collectively exhaustive as possible, with an explicit catch-all for inputs outside any defined category. This decision-then-act pattern foreshadows the ReAct architecture (Yao et al., 2023), covered in the agent design lesson.
# Role: request classifier. Classify the user request into exactly one category: - "data_extraction" — user wants structured data pulled from a document - "summarisation" — user wants a shorter version of provided content - "question_answer" — user is asking a factual question - "code_generation" — user wants code written or modified - "unknown" — the request does not fit any category above Return JSON only: {"type": "<one of the five values above>", "confidence": <0.0-1.0>} # Orchestrator logic (not part of the prompt): # if confidence < 0.6 -> fallback to a clarification step # if type === "unknown" -> route to a human escalation queue
Two engineering decisions matter more than the prompt text itself.
Structured output: force the router to produce a JSON object
with a fixed schema — {"type": "summarisation", "confidence": 0.91}
parses with a single call, where prose requires fragile string matching.
A confidence threshold: any router call below a configurable
threshold should not route — it should trigger a clarification or fallback
path. A wrong route is significantly more expensive than a delayed one.
A router classifies. A dispatcher acts on that classification. Keep these as separate concerns: the router is a pure function, input in, label out; the dispatcher owns the mapping from label to chain, the fallback rules, and the confidence thresholds. Mixing the two produces prompts that are hard to test and hard to change.
5. State and context across steps
Every chain step receives some state and produces some state. Managing that state carefully is what separates a production pipeline from a proof of concept that falls apart on edge cases.
What to pass forward
The default instinct — pass the full output of every previous step into every subsequent step — is wrong. It balloons the context window with every step, increases token cost, and degrades output quality by forcing the model to parse irrelevant prior content. The correct principle is minimum necessary context: each step receives only its direct predecessor's output plus any global context genuinely required. If a later step needs information from three steps back, that is a signal the intermediate steps should have preserved it explicitly in their structured output.
The state schema pattern
Define a shared state schema at the start of pipeline design, before writing any individual step prompt. The schema is a JSON object that accumulates data as it passes through the chain — each step reads from it, makes its contribution, and writes back. Steps do not communicate via prose; they communicate via fields.
// Initial state {"user_request": "...", "source_document": "..."} // After step 1 {...initial, "extracted_facts": [...]} // After step 2 {...prev, "ranked_facts": [...], "contradictions": [...]} // After step 3 {...prev, "draft": "...", "draft_score": 8} // Final state {"final_report": "...", "metadata": {"steps": 3, "iterations": 2}}
Context compression
For long pipelines over long documents, even minimum-necessary-context passing can approach token limits. The solution is a compression step — a dedicated prompt whose only job is to summarise accumulated state into a compact form that preserves everything downstream steps need. Khattab et al. (2023) address this systematically in DSPy, which compiles declarative LM pipelines by automatically optimising prompts and few-shot examples across steps.
6. Failure modes and error propagation
A chain is only as reliable as its weakest boundary. Failure modes in chained systems tend to be silent, delayed, and hard to attribute.
Three boundary failures
- Format contract violations. Step N promises JSON; step N+1 calls parse on it. A markdown fence or an apologetic preamble crashes the chain — not at step N where the error originated, but at N+1 where it is consumed. Always validate the output schema at every step boundary before passing it downstream. Libraries like Zod (TypeScript) or Pydantic (Python) make this trivial, and route failures to a repair prompt.
- Hallucination inheritance. A factual error at step 2 does not get corrected at step 3 unless step 3 explicitly validates against a ground-truth source. Insert grounding checks at any step producing factual claims — have it cite the source sentence for each claim.
- Error amplification in loops. If a generate-critique-revise loop's critic is poorly calibrated, the revision step acts on flawed feedback and can produce output worse than what it replaced. Evaluate the critic prompt independently, against known-good and known-bad drafts, before trusting it as a termination gate.
# Role: JSON repair. The following text was supposed to be valid JSON matching this schema: {schema} It is not valid JSON. Extract and repair it, returning only the corrected JSON — no commentary. If the content cannot be recovered, return {"error": "unrecoverable"}. Malformed input: {malformed_output} # This step runs only when JSON.parse throws. # Log all repair invocations — a high repair rate signals # a failing step prompt.
Log the full input and output of every chain step in production. A pipeline that fails at step 5 is only debuggable if you can inspect what steps 1 through 4 produced. Logging LLM calls is not optional overhead — it is the primary debugging tool for agentic systems.
7. When not to chain
Chaining is not universally better than a single well-crafted prompt. It has real costs — latency, token expense, engineering complexity, more failure surfaces — and those costs are not worth paying for tasks a single prompt handles reliably.
Use a chain when
- The task has multiple qualitatively different sub-tasks.
- The output of one step is verifiable independently.
- Reliability on the monolith is below your quality threshold.
- Error localisation matters.
Stay monolithic when
- The task is a single transformation.
- Latency is critical.
- The model is already consistent.
- The team cannot support the additional complexity of a multi-step system.
Do not build a chain when
- The task fits cleanly in one prompt — summarise this email, translate this sentence, classify this ticket. Adding a chain adds complexity without adding reliability.
- Latency is the primary constraint. Every step is at minimum one additional round-trip; under a two-second budget, a two-step chain is often infeasible.
- The output structure is simple and the model is already consistent on 99% of inputs — a formatting correction step is premature optimisation.
- The team cannot maintain the pipeline. A five-step chain with no logging, no schema validation, and no documented step contracts is harder to fix than a single monolithic prompt.
The practical decision rule: attempt the task with a single, well-engineered prompt first. Measure its reliability over a representative sample. If it meets your quality bar, ship it. If it doesn't, identify specifically which sub-task is failing — then chain only that sub-task out into its own step. Chaining is a tool for managing complexity at scale, not a default architecture. The best chain is the shortest chain that meets your reliability requirement.
Was this lesson useful?
FAQ
Frequently asked questions
Prompt chaining breaks a complex task into a sequence of simpler sub-tasks, each handled by a dedicated prompt, where the output of one step becomes part of the input of the next. The model never sees the whole problem at once — each call is constrained to a scope where it can be excellent rather than merely adequate.
The failure is predictable rather than random: instructions interfere with each other, attention dilutes across competing objectives, output format drifts, and an early mistake contaminates every downstream requirement. Decomposing the task isolates each responsibility so failures stay local and diagnosable.
Sequential chains pass each step's output to the next; branching chains route to independent sub-chains based on a classification step; looped chains iterate generate-critique-revise cycles until a quality bar is met; and map-reduce chains apply the same prompt to many segments before aggregating. A router in front can classify each input and dispatch it to the right pipeline.
Define a structured output contract for every step and validate it before the result is passed downstream. A cheap assertion layer between steps catches malformed output early — without it, format breaks cascade and hallucinated details get restated as fact by later steps.
When the task fits cleanly in one prompt, when latency is the primary constraint (every step adds a full LLM round-trip), when a single prompt already returns the correct schema on nearly all inputs, or when the team cannot maintain the added complexity. The practical rule: build the minimum viable chain, not the maximum imaginable one.
References
- Wu, T., Terry, M., & Cai, C. J. (2022). AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Prompts. CHI 2022. arXiv:2110.01691.
- Dohan, D., et al. (2022). Language Model Cascades. arXiv:2207.10342.
- Liu, N. F., et al. (2023). Lost in the Middle: How Language Models Use Long Contexts. TACL 12. arXiv:2307.03172.
- Madaan, A., et al. (2023). Self-Refine: Iterative Refinement with Self-Feedback. NeurIPS 2023.
- Khattab, O., et al. (2023). DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines. arXiv:2310.03714.
- Yao, S., et al. (2023). ReAct: Synergizing Reasoning and Acting in Language Models. ICLR 2023.
- Wei, J., et al. (2022). Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. NeurIPS 2022.
- Chase, H. (2022). LangChain [software]. github.com/langchain-ai/langchain.
Ready to test yourself?
10 questions drawn at random from the six lessons of Part I. No time limit. Share your score.