Chain-of-Thought and Self-Consistency Mechanism, Variants and Limits

Quick answer

Chain-of-thought is not a trick phrase. It is a structural intervention: the model's own reasoning steps become tokens in its context, and every later token — including the answer — is conditioned on them.

  • Seven variants with real empirical support, each targeting a different failure mode of basic CoT.
  • Self-consistency samples N chains and takes the majority — 10 to 18 points on math benchmarks, at N times the cost.
  • Reasoning models invert the advice. With extended thinking on, a CoT trigger in the user turn competes with the model's internal process.

Lessons 01 and 02 established the foundation: prompts shape probability distributions, and the system prompt is the highest-leverage layer in any production system. Both mentioned chain-of-thought as a core technique. This lesson examines it properly — mechanism, variants, limits, and the question that matters most in 2026: what happens to CoT when the model reasons internally before it speaks?

1. The mechanism, at the token level

A language model generates tokens left to right, one at a time. Each token is sampled from a distribution conditioned on everything before it — the entire context window, including everything the model has already produced in the current response. That is the architectural fact that makes chain-of-thought work: the model's own intermediate output becomes part of its context.

When a model writes "let me think through this step by step" and then produces reasoning steps, those steps are not decoration. They are tokens in the context window. Every subsequent token, the final answer included, is conditioned on them. The model is, in a precise sense, reading its own reasoning before it concludes.

The model cannot jump ahead to the answer without passing through intermediate token states that constrain what is likely next. This property — not any special reasoning faculty — is why externalising reasoning improves accuracy. On genuinely multi-step problems (logical inference, multi-document analysis, planning under constraints) the direct path produces materially higher error rates, because the model lacks the intermediate tokens that would pull its answer distribution toward the correct region.

The core mechanism

CoT works because each reasoning step is added to the context before the next step is generated. The model cannot "jump ahead" to the answer without passing through intermediate token states that constrain what tokens are likely next. This architectural property — not any special reasoning capability — is why externalising reasoning improves accuracy.

Direct answer vs. reasoning trace Example
# Without CoT — direct path
Q: A shop sells apples at £0.40 each and pears at £0.65
each. A customer buys 5 apples and 3 pears. What is
the total?
A: £3.95

# With CoT — reasoning trace
Q: A shop sells apples at £0.40 each and pears at £0.65
each. A customer buys 5 apples and 3 pears. What is
the total?
A: 5 apples × £0.40 = £2.00. 3 pears × £0.65 = £1.95.
Total = £2.00 + £1.95 = £3.95.

The arithmetic example is simple enough that both paths reach the same answer. On genuinely multi-step problems — logical inference, multi-document analysis, planning with constraints — the direct path produces significantly higher error rates because the model lacks the intermediate tokens that would constrain its answer distribution toward the correct region.

The reasoning trace is not an explanation of the answer. It is part of the input that produces the answer.
Why the order of generation matters

2. The original finding, and what it actually showed

Wei et al. (2022) introduced chain-of-thought prompting in a paper that remains among the most cited in the field. Prompting models to produce intermediate reasoning steps before a final answer improved benchmark performance by a large margin — often 40% or more — on models of sufficient scale. Two aspects of that result are routinely misread.

Wei et al. (2022) — key result

Chain-of-thought prompting significantly increases the ability of language models to perform complex reasoning — sometimes exceeding fine-tuned state-of-the-art models on arithmetic, commonsense, and symbolic reasoning tasks.

Scale dependence. CoT degrades performance on small models. In the original study the technique only reliably improved accuracy above roughly 100 billion parameters, and the effect was emergent — it appeared discontinuously at scale rather than improving gradually. The threshold has since moved a long way as training methods improved; 7B–13B models today show clear CoT benefits. The principle holds regardless: CoT needs a model with enough capacity to treat intermediate steps as meaningful constraints. Below that, it adds noise.

Task-type dependence. CoT improves compositional reasoning — tasks where discrete steps must be chained correctly. It offers little or nothing on retrieval ("what year did X happen?") or on single-step tasks. It is a targeted intervention, not a universal performance booster.

The paper introduced few-shot CoT, where each example shows both the question and an annotated step-by-step solution. Kojima et al. (2022) then showed that the far simpler zero-shot variant — appending "let's think step by step" — captures most of the benefit without any annotated examples at all.

3. Seven variants, and when to use each

Since 2022 a family of CoT variants has grown up, each targeting a different failure mode of the basic technique. The Schulhoff et al. (2024) survey catalogues them across 1,565 papers. Seven have strong empirical support and clear production use cases.

The simplest form. Append a trigger phrase to the prompt that instructs the model to reason before concluding. No examples required. Kojima et al. (2022) showed that the phrase "Let's think step by step" outperforms all alternatives tested, including more elaborate formulations — its brevity and directness appear to be part of its effectiveness.

Zero-shot CoT — trigger phrases Benchmark tested
— Highest-performing (Kojima et al. 2022)
Let's think step by step.

— Stronger for structured tasks
Think through this carefully, one step at a time.

— Stronger for causal / diagnostic reasoning
Before answering, work through the problem
from first principles. Show your reasoning.

Use when: no labelled examples are available, the task is novel or varied, or you need a fast baseline improvement with minimal prompt overhead. Zero-shot CoT is the right first experiment before investing in few-shot examples.

4. Self-consistency — sampling many paths

Wang et al. (2022) identified a fundamental limitation of greedy CoT: when the model generates a single reasoning chain, the quality of the final answer depends entirely on whether that particular chain happened to be correct. Errors early in the reasoning trace compound; the model has no mechanism to detect that it has gone wrong.

Self-consistency samples N independent reasoning chains from the same prompt at non-zero temperature and takes the majority answer. Different chains make different errors, so the majority vote is markedly more reliable than any single chain. Wang et al. measured gains of 17.9 points on GSM8K, 11.0 on SVAMP and 12.2 on AQuA against standard CoT. The improvement is largest where one correct answer can be reached by several valid routes.

Prompt

One question, one CoT-style prompt — no changes between samples.

Sample N

Generate N independent reasoning chains at non-zero temperature, typically N = 5.

Marginalise

Extract the final answer from each chain and take the majority vote across all N.

Output

The majority answer, discarding the individual chains that produced it.

Where self-consistency earns its cost

  • High-consequence classification (medical triage, legal risk assessment, fraud detection).
  • Multi-step quantitative reasoning where a single error would be costly.
  • Automated decision systems where human review is not available downstream.
  • Evaluation of other prompts — using self-consistency as the gold standard against which simpler prompts are compared.

N = 5

Where to start

0.7

Sampling temperature

×N

Cost and latency

The cost trade-off

Self-consistency multiplies API cost and latency by N. It is not a technique for every call — it is a targeted investment for high-stakes decisions where reliability justifies the bill. Start at N = 5 and measure. In most production tasks the majority answer stabilises between 5 and 10; going to 20 rarely beats 10 by anything meaningful and doubles the cost. Sample at temperature 0.7 unless the task is purely factual, where lower temperatures produce less diverse chains and weaken the whole method.

5. Tree of Thoughts — deliberate search

Standard CoT and self-consistency are both linear in time: they generate a sequence of tokens start to finish. Yao et al. (2023) introduced Tree of Thoughts as a different architecture — a search over a tree of reasoning states, with the model acting as both generator and evaluator at each node.

The model produces several candidate thoughts at each branch point. A separate evaluation step — another call, or the same model prompted as an evaluator — scores each candidate, and only the highest-scoring branches are expanded. On the Game of 24, GPT-4 with standard CoT solved 4% of problems; with ToT it solved 74%. Yao et al. also demonstrated ToT on creative writing with structural constraints and a crossword mini-puzzle benchmark.

Decompose

Break the problem into intermediate "thought" steps small enough to generate and evaluate independently.

Generate

At each branch point, produce K candidate next-thoughts — typically K = 3–5.

Evaluate

Score each candidate — another call, or the same model prompted as an evaluator.

Search

Expand only the highest-scoring branches, using a BFS or DFS policy over the tree.

Conclude

Once a branch reaches a complete solution, return it as the final answer.

Before you reach for it

ToT costs roughly N × K × depth model calls per task. A modest configuration — depth 3, branching factor 3, three evaluation calls per node — lands near 30 to 40 calls. That puts it firmly in offline territory.

  • Batch processing, research pipelines and long-horizon planning: yes.
  • Real-time, user-facing paths: no.
  • The deployment pattern that works is generating high-quality outputs offline, then caching them or using them as few-shot examples for a lighter online model. Treat ToT as a generation process whose outputs are assets, not as a runtime inference strategy.

6. Reasoning models change the equation

OpenAI's o1, then o3 and o4-mini, and Anthropic's Claude Sonnet 4.6 and Opus 4.6 with extended thinking, introduced a model category that changes one thing about the picture above: the reasoning chain is generated internally, before the visible output begins. These models reason in a hidden scratchpad, backtrack, explore alternatives, and only then produce a response. What you read is the conclusion of a process you never see.

Reasoning location

Standard: in the visible output. Reasoning model: in a hidden scratchpad.

CoT trigger

Standard: required to elicit the behaviour. Reasoning model: redundant, and often harmful.

Reasoning depth

Standard: proportional to the context you spend. Reasoning model: a configurable thinking budget.

Latency

Standard: low — reasoning tokens are generation tokens. Reasoning model: higher, thinking precedes output.

Prompt strategy

Standard: role + CoT trigger + format. Reasoning model: role + constraints + format, no trigger.

Self-consistency

Standard: valuable — each sample explores a different path. Reasoning model: less necessary, the internal search already does.

Anthropic's documentation states plainly that prompting for explicit step-by-step reasoning in the user turn is unnecessary with extended thinking, and may produce worse results than letting it run uninterrupted. OpenAI's guidance for the o-series says the same: avoid CoT-style instructions, spend the prompt on clear task specification instead.

Self-consistency's value also diminishes on reasoning models, because the internal search already explores multiple reasoning paths within a single generation. Sampling N responses still provides some gain, but the marginal return per additional sample is lower than on standard models. The cost-to-benefit trade-off shifts accordingly.

7. When not to use it

CoT gets applied reflexively to anything that looks hard. That is a mistake. Four conditions make it counterproductive.

Four conditions

  • Simple or single-step tasks. Look up a fact, format a string, extract a named entity — the intermediate steps are padding, not reasoning. Use zero-shot with a tight format spec.
  • Underpowered models. Below roughly 7B parameters, CoT can reduce accuracy by generating plausible-looking but wrong steps that condition the answer toward the wrong region. Test against your baseline; do not assume it helps.
  • Latency-critical paths. Real-time autocomplete, transcription post-processing, sub-second chat — these cannot absorb the extra tokens. Take the accuracy trade-off and keep CoT for batch work.
  • Reasoning models. The single most common mistake when migrating from standard models. Output looks more verbose; quality is often lower, because the user-turn instruction competes with the internal process rather than adding to it.

8. Decision guide — choosing a variant

The choice should be driven by the task type, the model, and the cost constraint. These are the common production scenarios and where each lands.

Task → technique

  • Multi-step arithmetic, cost is not the constraint. Self-consistency at N = 5–10 over zero-shot CoT.
  • Multi-step arithmetic, latency-sensitive. Few-shot CoT with two or three annotated examples.
  • Ambiguous classification with known failure modes. Contrastive CoT, with the correct and incorrect reasoning both labelled.
  • Domain analysis — legal, medical, financial. Step-back, then few-shot CoT.
  • Multi-variable planning or sequential decisions. Plan-and-solve, or Tabular CoT.
  • Novel task with no examples available. Analogical reasoning plus zero-shot CoT.
  • Complex constraint satisfaction, offline. Tree of Thoughts.
  • Any reasoning task on o-series or extended thinking. No CoT trigger — spend it on role, constraints and format.
  • Simple retrieval or formatting. No CoT — direct zero-shot with a format spec, on any model.

Every line above is a starting recommendation, not a guarantee. Build a test set of ten to fifteen representative inputs before committing to a variant. The technique that wins on your set is the right one, whatever the literature says about the general case. Task distributions vary; your data is the ground truth.

Lesson 04 moves from individual reasoning techniques to the architecture level: how to design and evaluate prompt chains, when to break a task across multiple model calls, and how to structure the handoffs so an error in one step does not propagate silently through the pipeline.

Was this lesson useful?

FAQ

Frequently asked questions

Chain-of-thought prompting instructs a language model to produce intermediate reasoning steps before its final answer. Because each step is added to the context window, it conditions subsequent token probabilities — including the answer — toward a more accurate region. Wei et al. (2022) introduced it and measured improvements of 40% or more on multi-step reasoning benchmarks.

References

  1. Wei, J., et al. (2022). Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. NeurIPS 2022. arXiv:2201.11903.
  2. Kojima, T., et al. (2022). Large Language Models are Zero-Shot Reasoners. NeurIPS 2022. arXiv:2205.11916.
  3. Schulhoff, S., et al. (2024). The Prompt Report: A Systematic Survey of Prompting Techniques. arXiv:2406.06608.
  4. Zheng, H. S., et al. (2023). Take a Step Back: Evoking Reasoning via Abstraction in Large Language Models. ICLR 2024. arXiv:2310.06117.
  5. Wang, L., et al. (2023). Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models. ACL 2023. arXiv:2305.04091.
  6. Jin, Z., et al. (2023). Tab-CoT: Zero-shot Tabular Chain of Thought. ACL Findings 2023. arXiv:2305.17812.
  7. Yasunaga, M., et al. (2023). Large Language Models as Analogical Reasoners. ICLR 2024. arXiv:2310.01714.
  8. Chia, Y. K., et al. (2023). Contrastive Chain-of-Thought Prompting. arXiv:2311.09277.
  9. Wang, X., et al. (2022). Self-Consistency Improves Chain of Thought Reasoning in Language Models. arXiv:2203.11171.
  10. Yao, S., et al. (2023). Tree of Thoughts: Deliberate Problem Solving with Large Language Models. arXiv:2305.10601.
  11. Anthropic. (2025). Extended Thinking — Claude API Documentation. docs.anthropic.com.
  12. OpenAI. (2025). Reasoning Models — Prompting Guidance. platform.openai.com/docs/guides/reasoning.

Ready to test yourself?

10 questions drawn at random from the six lessons of Part I. No time limit. Share your score.

Take the Quiz

Contact

Got a system to build?

If you're building a RAG pipeline, an AI agent workflow, or an MCP integration and want to compare notes, swap ideas, or just avoid the mistakes I've already made — reach out.

Or get the Playbook in your inbox — one email when a new lesson ships, and nothing else.

No noise, no spam. Unsubscribe anytime.