Prompt Chaining and Pipeline Design

1. What is prompt chaining?

Prompt chaining is the practice of breaking a complex task into a sequence of simpler sub-tasks, each handled by a dedicated prompt, where the output of one step becomes part of the input of the next. The model never sees the whole problem at once; it only sees the sub-problem it is currently responsible for.

The concept is explicit in Wu et al. (2022), who define a prompt chain as "a sequence of prompts where the output of one prompt is used as the input to the next."^#CHI Their AI Chains framing made these sequences transparent and controllable — an accurate mental model that foreshadowed today's DAG-based agent frameworks like LangGraph.^#LangChain

The anatomy of a single chain step is always the same regardless of how elaborate the overall pipeline becomes:

Dohan et al. (2022) formalise this further under the term language model cascades — probabilistic programs where each step conditions on the outputs of previous steps and each transition can incorporate deterministic logic.^#Cascade That framing is useful: a chain is not just a sequence of LLM calls, it is a program in which LLM calls are the operations and string-formatted state is the memory.

2. Why monolithic prompts break at scale

Before reaching for a chain, it is worth understanding precisely why a single prompt fails on complex tasks. The failure is not random — it follows predictable patterns.

Attention diffusion

Transformer attention is distributed across all tokens in the context window. When a prompt contains a task description, a long document, several examples, formatting rules, and output constraints simultaneously, the model's effective attention on any individual component decreases. Instructions near the middle of a long prompt are consistently under-weighted relative to those at the beginning and end — a finding confirmed across multiple benchmarks.^#Middle A chain localises each step's attention to only the tokens that matter for that step.

Format contract conflicts

Complex tasks often require the model to simultaneously reason, retrieve, classify, and format. These demands impose conflicting constraints. A model asked to "analyse this contract, identify liability clauses, check each against a rubric, and output a JSON report with a supporting narrative" is being asked to write in three registers — analytical prose, structured data, and evaluative commentary — inside a single generation pass. Reliability degrades because the model cannot fully commit to any single format.

Error compounding

In a monolithic prompt, an incorrect inference in the first half of the response tends to propagate. Because the model generates autoregressively, a flawed claim becomes context for everything that follows. In a chain, errors are local to the step in which they occur and can be caught at the boundary before they contaminate downstream steps.

3. The four chain architectures

Four architectures cover the majority of real-world chaining needs. They are not mutually exclusive — production pipelines frequently combine all four — but it is useful to understand each in isolation first.

Sequential chains

The sequential chain is the canonical pattern: a fixed, ordered pipeline where each step has one job. It is the right default for any task that can be expressed as a series of transformations on text.

A practical example: producing a competitive analysis from a set of raw research notes. The pipeline might be: (1) extract structured facts from each source, (2) normalise entities across sources, (3) identify contradictions, (4) rank findings by relevance, (5) draft the analysis. Each step is small, verifiable, and independently debuggable.

Branching chains

A branching chain uses a classification or routing step whose output determines which sub-chain runs next. This is the correct architecture when the same input type can require qualitatively different handling depending on its content.

The critical requirement is that the classification step produces a finite, deterministic set of labels that map cleanly to downstream chains. If the label space is fuzzy, the branch logic becomes unreliable. Constrain the classifier to a fixed vocabulary and use structured output to enforce it.

Looped / iterative chains

A looped chain passes the output of a step back to the same step (or to a preceding step) until a stopping condition is met. The canonical use case is the generate–critique–revise loop: one prompt drafts content, a second evaluates it against a rubric, and the draft is revised until it passes or a maximum iteration count is reached.

Madaan et al. (2023) formalise this as Self-Refine — a model iteratively improves its own output using feedback generated by itself.^#Refine The key finding is that a model acting as critic and reviser in separate, dedicated prompts outperforms a model asked to produce a final answer in a single pass, even when total token budget is held constant.

Define the termination condition before you write the loop. The three standard options are: a pass/fail gate (a validator step returns pass: true), a fixed iteration cap (run at most N times regardless of quality), or a diminishing-returns threshold (stop when the score delta between iterations falls below ε). In production, always combine a quality gate with a hard cap — an infinite loop is a live incident waiting to happen.

Map-reduce chains

When the input exceeds what a single context window can process reliably — or when independent parallel processing is desirable — the map-reduce pattern applies. The input is split into chunks, each chunk is processed by the same prompt (the map step), and a final prompt merges the results (the reduce step).

The reduce step is the hardest part to get right. A naive "summarise all summaries" reduce prompt produces flattened, generic output. A better approach is to have the reduce step perform a specific aggregation operation — ranking, contradiction detection, theme extraction — rather than free-form merging.

4. The router and dispatcher pattern

The router pattern deserves its own section because it appears at the entry point of virtually every non-trivial multi-chain system and its quality determines the reliability of everything downstream. A router is a prompt (or a lightweight function wrapping a prompt) that takes an incoming input and decides which chain should handle it.

The temptation is to write a router that tries to understand the input deeply. Resist it. A router's only job is classification into a finite set of categories. The categories should be mutually exclusive, and as collectively exhaustive as possible — with an explicit catch-all for inputs that fall outside any defined category — and mapped directly to named chains in your orchestration layer.

This decision-then-act pattern foreshadows the ReAct architecture (Yao et al., 2023), covered in the agent design lesson.^#ReAct

Two engineering decisions matter more than the prompt text itself. First, structured output: force the router to produce a JSON object with a fixed schema. A router that outputs prose like "this looks like a summarisation request" requires fragile string parsing. One that outputs {"type": "summarisation", "confidence": 0.91} can be processed with a single JSON.parse call. Second, a confidence threshold: any router call with confidence below a configurable threshold should not route — it should trigger a clarification or fallback path. A wrong route is significantly more expensive than a delayed route.

5. State and context across steps

Every chain step receives some state and produces some state. Managing that state carefully is what separates a production pipeline from a proof of concept that falls apart on edge cases.

What to pass forward

The default instinct is to pass the full output of every previous step into every subsequent step. This instinct is wrong. It leads to context windows that balloon with every step, increases token cost, and — crucially — degrades output quality by forcing the model to parse through irrelevant prior content to find what it actually needs.

The correct principle is minimum necessary context: each step receives only the output of its direct predecessor plus any global context that is genuinely required (e.g., the original user intent, a shared rubric, a fixed set of constraints). If a later step needs information from three steps back, that is a signal the intermediate steps should have preserved it explicitly in their structured output.

The state schema pattern

Define a shared state schema at the start of pipeline design, before writing any individual step prompt. The schema is a JSON object that accumulates data as it passes through the chain. Each step reads from it, makes its contribution, and writes back. Steps do not communicate via prose — they communicate via fields.

Context compression

For long pipelines processing long documents, even minimum-necessary-context passing can produce states that approach token limits. The solution is a compression step — a dedicated prompt whose only job is to summarise the accumulated state into a compact form that preserves all information required by downstream steps while discarding everything else. Khattab et al. (2023) address this systematically in DSPy, which compiles declarative LM pipelines by automatically optimising prompts and few-shot examples across steps — removing the need for hand-crafted inter-step instructions.^#DSPy

6. Failure modes and error propagation

A chain is only as reliable as its weakest boundary. Failure modes in chained systems are different from — and often more insidious than — failures in single-prompt systems. They tend to be silent, delayed, and hard to attribute.

Format contract violations

Step N promises to return JSON. Step N+1 calls JSON.parse on the output. If the model wraps its output in a markdown code fence, adds an apologetic preamble, or returns malformed JSON, the parse fails and the entire chain crashes — not at step N where the error originated, but at step N+1 where it is consumed. Always validate the output schema at every step boundary before passing it downstream. Libraries like Zod (TypeScript) or Pydantic (Python) make this trivial. If validation fails, route to a repair prompt that attempts to re-format the malformed output before re-raising.

Hallucination inheritance

A factual error generated at step 2 does not get corrected at step 3 unless step 3 explicitly validates against a ground-truth source. In most pipelines it does not — it gets treated as authoritative input and built upon. A wrong entity name in an extraction step becomes a wrong entity name in a summary, which becomes a wrong entity name in a final report. Insert grounding checks at any step that produces factual claims: have the step cite the specific sentence from the source document that supports each claim. Unsupported claims fail the boundary validation and trigger a retry or a flagged output.

Error amplification in loops

Looped chains are particularly susceptible to amplification. If the critique step in a generate–critique–revise loop is poorly calibrated — too lenient, too harsh, or inconsistent — the revision step acts on flawed feedback and may produce output that is worse than what it replaced. Always evaluate the critic prompt independently before wiring it into a loop. Run it against a suite of known-good and known-bad drafts to verify its accuracy before trusting it as a termination gate.

7. When not to chain

Chaining is not universally better than a single well-crafted prompt. It has real costs — latency, token expense, engineering complexity, and more failure surfaces — and those costs are not worth paying for tasks that a single prompt handles reliably.

Do not build a chain when:

• The task fits cleanly in one prompt. Summarise this email. Translate this sentence. Classify this support ticket. These are single-step tasks. Adding a chain adds complexity without adding reliability.
• Latency is the primary constraint. Every step in a chain is at minimum one additional LLM round-trip. For a latency budget of under two seconds, a two-step chain is often infeasible at typical inference speeds. A single, carefully crafted prompt will always be faster.
• The output structure is simple and the model is consistent. If you have verified empirically that a single prompt returns the exact schema you need on 99% of inputs, adding a formatting correction step is premature optimisation.
• You cannot maintain the pipeline. A chain of five steps with no logging, no schema validation, and no documented step contracts is harder to fix than a single monolithic prompt. If your team cannot maintain the added complexity, the chain will degrade silently over time as models update and prompt semantics shift.

The practical decision rule: attempt the task with a single, well-engineered prompt first. Measure its reliability over a representative sample of inputs. If it meets your quality bar, ship it. If it doesn't, identify specifically which sub-task is failing — then chain only that sub-task out into its own step. Build the minimum viable chain, not the maximum imaginable one.

Chaining is a tool for managing complexity at scale, not a default architecture. The best chain is the shortest chain that meets your reliability requirement.

References

1. #CHI Wu, T., Terry, M., & Cai, C. J. (2022). AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Prompts. CHI 2022. arXiv:2110.01691
2. #Cascade Dohan, D., Xu, W., Lewkowycz, A., Austin, J., Bieber, D., Lopes, R. G., Wu, Y., Michalewski, H., Saurous, R. A., Sohl-Dickstein, J., Murphy, K., & Sutton, C. (2022). Language Model Cascades. arXiv preprint arXiv:2207.10342. arxiv.org
3. #Middle Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2023). Lost in the Middle: How Language Models Use Long Contexts. Transactions of the Association for Computational Linguistics, 12, 157–173. arxiv.org
4. #Refine Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., Gupta, S., Majumder, B. P., Hermann, K., Welleck, S., Yazdanbakhsh, A., & Clark, P. (2023). Self-Refine: Iterative Refinement with Self-Feedback. NeurIPS 2023. arxiv.org
5. #DSPy Khattab, O., Singhvi, A., Maheshwari, P., Zhang, Z., Shrivastava, K., Sharma, S., Colby Dodds, Z., Strubell, E., Naous, T., Gupta, R., Lerer, A., Zaharia, M., & Potts, C. (2023). DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines. arXiv preprint arXiv:2310.03714. arxiv.org
6. #ReAct Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., & Cao, Y. (2023). ReAct: Synergizing Reasoning and Acting in Language Models. ICLR 2023. arxiv.org
7. #CoT Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E. H., Le, Q. V., & Zhou, D. (2022). Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. NeurIPS 2022. arxiv.org
8. #LangChain Chase, H. (2022). LangChain [software]. GitHub. github.com/langchain-ai/langchain