Quick answer
Chain-of-thought is not a trick phrase. It is a structural intervention: the model's own reasoning steps become tokens in its context, and every later token — including the answer — is conditioned on them.
- Seven variants with real empirical support, each targeting a different failure mode of basic CoT.
- Self-consistency samples N chains and takes the majority — 10 to 18 points on math benchmarks, at N times the cost.
- Reasoning models invert the advice. With extended thinking on, a CoT trigger in the user turn competes with the model's internal process.
Lessons 01 and 02 established the foundation: prompts shape probability distributions, and the system prompt is the highest-leverage layer in any production system. Both mentioned chain-of-thought as a core technique. This lesson examines it properly — mechanism, variants, limits, and the question that matters most in 2026: what happens to CoT when the model reasons internally before it speaks?
1. The mechanism, at the token level
A language model generates tokens left to right, one at a time. Each token is sampled from a distribution conditioned on everything before it — the entire context window, including everything the model has already produced in the current response. That is the architectural fact that makes chain-of-thought work: the model's own intermediate output becomes part of its context.
When a model writes "let me think through this step by step" and then produces reasoning steps, those steps are not decoration. They are tokens in the context window. Every subsequent token, the final answer included, is conditioned on them. The model is, in a precise sense, reading its own reasoning before it concludes.
The model cannot jump ahead to the answer without passing through intermediate token states that constrain what is likely next. This property — not any special reasoning faculty — is why externalising reasoning improves accuracy. On genuinely multi-step problems (logical inference, multi-document analysis, planning under constraints) the direct path produces materially higher error rates, because the model lacks the intermediate tokens that would pull its answer distribution toward the correct region.
The core mechanism
CoT works because each reasoning step is added to the context before the next step is generated. The model cannot "jump ahead" to the answer without passing through intermediate token states that constrain what tokens are likely next. This architectural property — not any special reasoning capability — is why externalising reasoning improves accuracy.
# Without CoT — direct path Q: A shop sells apples at £0.40 each and pears at £0.65 each. A customer buys 5 apples and 3 pears. What is the total? A: £3.95 # With CoT — reasoning trace Q: A shop sells apples at £0.40 each and pears at £0.65 each. A customer buys 5 apples and 3 pears. What is the total? A: 5 apples × £0.40 = £2.00. 3 pears × £0.65 = £1.95. Total = £2.00 + £1.95 = £3.95.
The arithmetic example is simple enough that both paths reach the same answer. On genuinely multi-step problems — logical inference, multi-document analysis, planning with constraints — the direct path produces significantly higher error rates because the model lacks the intermediate tokens that would constrain its answer distribution toward the correct region.
The reasoning trace is not an explanation of the answer. It is part of the input that produces the answer.
2. The original finding, and what it actually showed
Wei et al. (2022) introduced chain-of-thought prompting in a paper that remains among the most cited in the field. Prompting models to produce intermediate reasoning steps before a final answer improved benchmark performance by a large margin — often 40% or more — on models of sufficient scale. Two aspects of that result are routinely misread.
Wei et al. (2022) — key result
Chain-of-thought prompting significantly increases the ability of language models to perform complex reasoning — sometimes exceeding fine-tuned state-of-the-art models on arithmetic, commonsense, and symbolic reasoning tasks.
Scale dependence. CoT degrades performance on small models. In the original study the technique only reliably improved accuracy above roughly 100 billion parameters, and the effect was emergent — it appeared discontinuously at scale rather than improving gradually. The threshold has since moved a long way as training methods improved; 7B–13B models today show clear CoT benefits. The principle holds regardless: CoT needs a model with enough capacity to treat intermediate steps as meaningful constraints. Below that, it adds noise.
Task-type dependence. CoT improves compositional reasoning — tasks where discrete steps must be chained correctly. It offers little or nothing on retrieval ("what year did X happen?") or on single-step tasks. It is a targeted intervention, not a universal performance booster.
The paper introduced few-shot CoT, where each example shows both the question and an annotated step-by-step solution. Kojima et al. (2022) then showed that the far simpler zero-shot variant — appending "let's think step by step" — captures most of the benefit without any annotated examples at all.
3. Seven variants, and when to use each
Since 2022 a family of CoT variants has grown up, each targeting a different failure mode of the basic technique. The Schulhoff et al. (2024) survey catalogues them across 1,565 papers. Seven have strong empirical support and clear production use cases.
The simplest form. Append a trigger phrase to the prompt that instructs the model to reason before concluding. No examples required. Kojima et al. (2022) showed that the phrase "Let's think step by step" outperforms all alternatives tested, including more elaborate formulations — its brevity and directness appear to be part of its effectiveness.
— Highest-performing (Kojima et al. 2022) Let's think step by step. — Stronger for structured tasks Think through this carefully, one step at a time. — Stronger for causal / diagnostic reasoning Before answering, work through the problem from first principles. Show your reasoning.
Use when: no labelled examples are available, the task is novel or varied, or you need a fast baseline improvement with minimal prompt overhead. Zero-shot CoT is the right first experiment before investing in few-shot examples.
Each example shows both the question and an annotated step-by-step solution, conditioning the model on the reasoning style you want, not just the trigger to reason.
Question: A company's revenue grew from £2.4M to £3.1M
year-over-year. What was the percentage growth, rounded
to one decimal place?
Reasoning:
Growth = £3.1M − £2.4M = £0.7M
Percentage = (0.7 / 2.4) × 100 = 29.166...%
Rounded to one decimal place: 29.2%
Answer: 29.2%
---
Question: {{your question here}}
Reasoning:
Use when: the task has a consistent structure and you can afford one or two annotated examples in the prompt — the annotation locks in the reasoning style, not just the answer format. Example quality is critical. An example with a flawed reasoning trace — even one that reaches the correct answer — conditions the model toward similar flawed reasoning in the target case. Always verify the logical correctness of each step in your few-shot examples, not just the final answer.
Zheng et al. (2023) introduced step-back prompting: before solving the specific instance, the model first abstracts the general principle or question that governs it, then applies that principle to the specific case.
— Phase 1: abstraction Before analysing this specific contract clause, identify: What general legal principle governs this type of clause? What is the standard risk framework for evaluating it? — Phase 2: application (feeds on Phase 1 output) Now apply those principles to the following clause and identify the specific risks for a SaaS vendor: {{clause text}}
Use when: the task requires applying general domain knowledge to specific cases — legal analysis, medical diagnosis reasoning, technical root-cause analysis — where grounding the model in the right abstraction first measurably improves the specific answer.
Wang et al. (2023) identified a step-order failure mode in zero-shot CoT — the model sometimes reasons and acts in the same breath, skipping planning entirely. Plan-and-solve prompting separates the two explicitly: devise the plan first, then execute it.
Let's first understand the problem and devise a plan to solve it. Then, let's carry out the plan step by step and solve the problem.
Use when: the task involves multi-step planning or sequential decisions, where an unplanned reasoning trace tends to skip steps or reason out of order.
Jin et al. (2023) structure the reasoning trace itself as a table, forcing each intermediate result into its own cell rather than free-form prose.
Reason through the problem using a table with three columns: | Step | Reasoning | Intermediate result | Complete each row before moving to the next. After the table, state your final answer in one sentence.
Use when: reasoning chains tend to become long and lose coherence in free-form prose — especially effective for financial analysis, multi-variable optimisation, and eligibility or compliance reasoning.
Yasunaga et al. (2023) prompt the model to generate its own analogous, more familiar problem before tackling the real one, using the self-generated analogy as scaffolding for the actual reasoning.
Before solving this problem: 1. Identify a simpler or more familiar problem that has the same underlying structure. 2. Describe how you would solve that analogous problem. 3. Map that solution approach onto the current problem and solve it.
Use when: the task is genuinely novel or atypical, and no annotated few-shot examples exist to draw on — the model generates its own scaffold instead of being handed one.
Chia et al. (2023) show a labelled pair of reasoning traces — one flawed, one correct — for the same or a similar question, so the model learns to recognise and avoid the specific failure pattern rather than reasoning from scratch.
Question: Is 17 a prime number? ✗ Incorrect reasoning (labelled) 17 is odd. Odd numbers are prime. Therefore 17 is prime. Error: the premise "odd numbers are prime" is false (9, 15, 25…) ✓ Correct reasoning (labelled) 17 is only divisible by 1 and itself. Check: 17 ÷ 2 = 8.5, 17 ÷ 3 = 5.67, 17 ÷ 4 = 4.25 — none divide evenly. No factor exists between 1 and √17 ≈ 4.12. Therefore 17 is prime.
Use when: the task has common failure modes or systematic error patterns — classification tasks with easily confused categories, logical inference with a common fallacy, data extraction where a specific field is routinely misidentified.
4. Self-consistency — sampling many paths
Wang et al. (2022) identified a fundamental limitation of greedy CoT: when the model generates a single reasoning chain, the quality of the final answer depends entirely on whether that particular chain happened to be correct. Errors early in the reasoning trace compound; the model has no mechanism to detect that it has gone wrong.
Self-consistency samples N independent reasoning chains from the same prompt at non-zero temperature and takes the majority answer. Different chains make different errors, so the majority vote is markedly more reliable than any single chain. Wang et al. measured gains of 17.9 points on GSM8K, 11.0 on SVAMP and 12.2 on AQuA against standard CoT. The improvement is largest where one correct answer can be reached by several valid routes.
Prompt
One question, one CoT-style prompt — no changes between samples.
Sample N
Generate N independent reasoning chains at non-zero temperature, typically N = 5.
Marginalise
Extract the final answer from each chain and take the majority vote across all N.
Output
The majority answer, discarding the individual chains that produced it.
Where self-consistency earns its cost
- High-consequence classification (medical triage, legal risk assessment, fraud detection).
- Multi-step quantitative reasoning where a single error would be costly.
- Automated decision systems where human review is not available downstream.
- Evaluation of other prompts — using self-consistency as the gold standard against which simpler prompts are compared.
N = 5
Where to start
0.7
Sampling temperature
×N
Cost and latency
The cost trade-off
Self-consistency multiplies API cost and latency by N. It is not a technique for every call — it is a targeted investment for high-stakes decisions where reliability justifies the bill. Start at N = 5 and measure. In most production tasks the majority answer stabilises between 5 and 10; going to 20 rarely beats 10 by anything meaningful and doubles the cost. Sample at temperature 0.7 unless the task is purely factual, where lower temperatures produce less diverse chains and weaken the whole method.
5. Tree of Thoughts — deliberate search
Standard CoT and self-consistency are both linear in time: they generate a sequence of tokens start to finish. Yao et al. (2023) introduced Tree of Thoughts as a different architecture — a search over a tree of reasoning states, with the model acting as both generator and evaluator at each node.
The model produces several candidate thoughts at each branch point. A separate evaluation step — another call, or the same model prompted as an evaluator — scores each candidate, and only the highest-scoring branches are expanded. On the Game of 24, GPT-4 with standard CoT solved 4% of problems; with ToT it solved 74%. Yao et al. also demonstrated ToT on creative writing with structural constraints and a crossword mini-puzzle benchmark.
Decompose
Break the problem into intermediate "thought" steps small enough to generate and evaluate independently.
Generate
At each branch point, produce K candidate next-thoughts — typically K = 3–5.
Evaluate
Score each candidate — another call, or the same model prompted as an evaluator.
Search
Expand only the highest-scoring branches, using a BFS or DFS policy over the tree.
Conclude
Once a branch reaches a complete solution, return it as the final answer.
Before you reach for it
ToT costs roughly N × K × depth model calls per task. A modest configuration — depth 3, branching factor 3, three evaluation calls per node — lands near 30 to 40 calls. That puts it firmly in offline territory.
- Batch processing, research pipelines and long-horizon planning: yes.
- Real-time, user-facing paths: no.
- The deployment pattern that works is generating high-quality outputs offline, then caching them or using them as few-shot examples for a lighter online model. Treat ToT as a generation process whose outputs are assets, not as a runtime inference strategy.
6. Reasoning models change the equation
OpenAI's o1, then o3 and o4-mini, and Anthropic's Claude Sonnet 4.6 and Opus 4.6 with extended thinking, introduced a model category that changes one thing about the picture above: the reasoning chain is generated internally, before the visible output begins. These models reason in a hidden scratchpad, backtrack, explore alternatives, and only then produce a response. What you read is the conclusion of a process you never see.
Reasoning location
Standard: in the visible output. Reasoning model: in a hidden scratchpad.
CoT trigger
Standard: required to elicit the behaviour. Reasoning model: redundant, and often harmful.
Reasoning depth
Standard: proportional to the context you spend. Reasoning model: a configurable thinking budget.
Latency
Standard: low — reasoning tokens are generation tokens. Reasoning model: higher, thinking precedes output.
Prompt strategy
Standard: role + CoT trigger + format. Reasoning model: role + constraints + format, no trigger.
Self-consistency
Standard: valuable — each sample explores a different path. Reasoning model: less necessary, the internal search already does.
Anthropic's documentation states plainly that prompting for explicit step-by-step reasoning in the user turn is unnecessary with extended thinking, and may produce worse results than letting it run uninterrupted. OpenAI's guidance for the o-series says the same: avoid CoT-style instructions, spend the prompt on clear task specification instead.
Self-consistency's value also diminishes on reasoning models, because the internal search already explores multiple reasoning paths within a single generation. Sampling N responses still provides some gain, but the marginal return per additional sample is lower than on standard models. The cost-to-benefit trade-off shifts accordingly.
7. When not to use it
CoT gets applied reflexively to anything that looks hard. That is a mistake. Four conditions make it counterproductive.
Four conditions
- Simple or single-step tasks. Look up a fact, format a string, extract a named entity — the intermediate steps are padding, not reasoning. Use zero-shot with a tight format spec.
- Underpowered models. Below roughly 7B parameters, CoT can reduce accuracy by generating plausible-looking but wrong steps that condition the answer toward the wrong region. Test against your baseline; do not assume it helps.
- Latency-critical paths. Real-time autocomplete, transcription post-processing, sub-second chat — these cannot absorb the extra tokens. Take the accuracy trade-off and keep CoT for batch work.
- Reasoning models. The single most common mistake when migrating from standard models. Output looks more verbose; quality is often lower, because the user-turn instruction competes with the internal process rather than adding to it.
8. Decision guide — choosing a variant
The choice should be driven by the task type, the model, and the cost constraint. These are the common production scenarios and where each lands.
Task → technique
- Multi-step arithmetic, cost is not the constraint. Self-consistency at N = 5–10 over zero-shot CoT.
- Multi-step arithmetic, latency-sensitive. Few-shot CoT with two or three annotated examples.
- Ambiguous classification with known failure modes. Contrastive CoT, with the correct and incorrect reasoning both labelled.
- Domain analysis — legal, medical, financial. Step-back, then few-shot CoT.
- Multi-variable planning or sequential decisions. Plan-and-solve, or Tabular CoT.
- Novel task with no examples available. Analogical reasoning plus zero-shot CoT.
- Complex constraint satisfaction, offline. Tree of Thoughts.
- Any reasoning task on o-series or extended thinking. No CoT trigger — spend it on role, constraints and format.
- Simple retrieval or formatting. No CoT — direct zero-shot with a format spec, on any model.
Every line above is a starting recommendation, not a guarantee. Build a test set of ten to fifteen representative inputs before committing to a variant. The technique that wins on your set is the right one, whatever the literature says about the general case. Task distributions vary; your data is the ground truth.
Lesson 04 moves from individual reasoning techniques to the architecture level: how to design and evaluate prompt chains, when to break a task across multiple model calls, and how to structure the handoffs so an error in one step does not propagate silently through the pipeline.
Was this lesson useful?
FAQ
Frequently asked questions
Chain-of-thought prompting instructs a language model to produce intermediate reasoning steps before its final answer. Because each step is added to the context window, it conditions subsequent token probabilities — including the answer — toward a more accurate region. Wei et al. (2022) introduced it and measured improvements of 40% or more on multi-step reasoning benchmarks.
No. CoT needs a model with enough capacity to use intermediate steps as real constraints. On very small models it can reduce accuracy by generating plausible-sounding but incorrect reasoning that biases the final answer. On modern instruction-tuned models of roughly 7B parameters and up, it generally helps on compositional reasoning tasks.
Self-consistency, introduced by Wang et al. (2022), samples N independent reasoning chains from the same prompt at non-zero temperature and takes the majority answer. Because different chains make different errors, the majority vote is markedly more reliable than any single chain — 10 to 18 percentage points on math benchmarks against standard CoT.
No. Reasoning models perform chain-of-thought internally in a hidden scratchpad. Telling them to think step by step in the user turn competes with that internal process and can lower output quality. Spend the prompt on role, constraints and output format instead.
Standard CoT generates one linear reasoning chain start to finish. Tree of Thoughts searches a tree of reasoning states — the model generates several candidate thoughts at each branch point, scores them, and expands only the most promising. It is far more powerful and costs 30 to 40 model calls per task, which puts it in offline batch processing rather than real-time paths.
References
- Wei, J., et al. (2022). Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. NeurIPS 2022. arXiv:2201.11903.
- Kojima, T., et al. (2022). Large Language Models are Zero-Shot Reasoners. NeurIPS 2022. arXiv:2205.11916.
- Schulhoff, S., et al. (2024). The Prompt Report: A Systematic Survey of Prompting Techniques. arXiv:2406.06608.
- Zheng, H. S., et al. (2023). Take a Step Back: Evoking Reasoning via Abstraction in Large Language Models. ICLR 2024. arXiv:2310.06117.
- Wang, L., et al. (2023). Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models. ACL 2023. arXiv:2305.04091.
- Jin, Z., et al. (2023). Tab-CoT: Zero-shot Tabular Chain of Thought. ACL Findings 2023. arXiv:2305.17812.
- Yasunaga, M., et al. (2023). Large Language Models as Analogical Reasoners. ICLR 2024. arXiv:2310.01714.
- Chia, Y. K., et al. (2023). Contrastive Chain-of-Thought Prompting. arXiv:2311.09277.
- Wang, X., et al. (2022). Self-Consistency Improves Chain of Thought Reasoning in Language Models. arXiv:2203.11171.
- Yao, S., et al. (2023). Tree of Thoughts: Deliberate Problem Solving with Large Language Models. arXiv:2305.10601.
- Anthropic. (2025). Extended Thinking — Claude API Documentation. docs.anthropic.com.
- OpenAI. (2025). Reasoning Models — Prompting Guidance. platform.openai.com/docs/guides/reasoning.
Ready to test yourself?
10 questions drawn at random from the six lessons of Part I. No time limit. Share your score.