Chain-of-Thought and Self-Consistency

1. The mechanism: why CoT works at the token level

A language model generates tokens left to right, one at a time. Each token is sampled from a probability distribution conditioned on everything that came before it — the entire context window, including everything the model has already produced in the current response. This is the key architectural fact that makes chain-of-thought work: the model's own intermediate output becomes part of its context.

When a model produces "Let me think through this step by step" and then writes out reasoning steps, those steps are not decoration. They are tokens in the context window. Every subsequent token — including the final answer — is conditioned on them. The model is, in a precise sense, reading its own reasoning before it concludes.

Compare the two generation paths below. The first goes directly from question to answer; the second passes through an intermediate reasoning trace before arriving at the answer. The intermediate trace changes the conditional distribution over the answer tokens — the tokens that are likely to follow a correct reasoning chain are different from those likely to follow the raw question alone.

The arithmetic example is simple enough that both paths reach the same answer. On genuinely multi-step problems — logical inference, multi-document analysis, planning with constraints — the direct path produces significantly higher error rates because the model lacks the intermediate tokens that would constrain its answer distribution toward the correct region.

2. The original finding and what it actually showed

Wei et al. (2022) introduced chain-of-thought prompting in a paper that remains one of the most cited in the field.^#Wei They showed that prompting models to produce intermediate reasoning steps before a final answer improved performance on those benchmarks by a large margin — often 40% or more — on models of sufficient scale.

Two aspects of this finding are frequently misunderstood and worth stating precisely.

Scale dependence. CoT prompting degrades performance on small models. Wei et al. found that the technique only reliably improves accuracy above approximately 100 billion parameters in the original study. The effect was "emergent" — it appeared discontinuously at scale rather than improving gradually. This threshold has since shifted substantially as training methods improved: much smaller models today (7B–13B parameters) show CoT benefits, but the general principle holds — CoT requires a model with sufficient capacity to leverage intermediate reasoning. On very small models, CoT can actually hurt by adding confusing noise.^#Wei

Task type dependence. CoT improves performance on tasks that require compositional reasoning — tasks where a series of discrete steps must be chained correctly to produce the answer. It offers little or no benefit on tasks that are essentially retrieval (what year did X happen?) or tasks where reasoning is single-step. The technique is not a universal performance booster; it is a targeted intervention for a specific class of tasks.

The paper introduced few-shot CoT — each example showed both the question and an annotated step-by-step solution. Kojima et al. (2022) subsequently demonstrated that the simpler zero-shot variant — appending "Let's think step by step" to a prompt — was sufficient to elicit most of the benefit without requiring annotated examples.^#ZeroShot

3. Seven CoT variants — and when to use each

Since Wei et al. (2022), a family of CoT variants has been developed, each targeting a different failure mode of basic chain-of-thought. The Schulhoff et al. (2024) survey catalogues these systematically across 1,565 papers.^#Survey Seven variants have strong empirical support and clear production use cases.

Zero-shot CoT

The simplest form. Append a trigger phrase to the prompt that instructs the model to reason before concluding. No examples required. Kojima et al. (2022) showed that the phrase "Let's think step by step" outperforms all alternatives tested, including more elaborate formulations — its brevity and directness appear to be part of its effectiveness.^#ZeroShot

— Highest-performing (Kojima et al. 2022)
  Let's think step by step.
  
  — Stronger for structured tasks
  Think through this carefully, one step at a time.
  
  — Stronger for causal / diagnostic reasoning
  Before answering, work through the problem
  from first principles. Show your reasoning.

Use when: no labelled examples are available, the task is novel or varied, or you need a fast baseline improvement with minimal prompt overhead. Zero-shot CoT is the right first experiment before investing in few-shot examples.

Few-shot CoT

The original formulation from Wei et al. Each few-shot example includes not just the input and the final answer, but the full reasoning trace that connects them. The model learns both what to produce and how to reason to get there.

Question: A company's revenue grew from £2.4M to £3.1M
  year-over-year. What was the percentage growth, rounded
  to one decimal place?
  
  Reasoning:
  Growth = £3.1M − £2.4M = £0.7M
  Percentage = (0.7 / 2.4) × 100 = 29.166...%
  Rounded to one decimal place: 29.2%
  
  Answer: 29.2%
  
  ---
  
  Question: {{your question here}}
  Reasoning:

Use when: the task has a consistent structure, you have access to correct examples with known reasoning paths, and zero-shot CoT produces inconsistent reasoning styles across runs. Few-shot CoT calibrates both the format and the reasoning pattern simultaneously.

Example quality is critical. An example with a flawed reasoning trace — even one that reaches the correct answer — conditions the model toward similar flawed reasoning in the target case. Always verify the logical correctness of each step in your few-shot examples, not just the final answer.

Step-back prompting

Introduced by Zheng et al. (2023), step-back prompting adds a preliminary abstraction step before the task-specific reasoning.^#StepBack Before solving the specific problem, the model is asked to identify the underlying principle or category of problem it belongs to. This activates higher-level knowledge from pretraining before the model applies it to the specific case.

— Phase 1: abstraction
Before analysing this specific contract clause, identify:
What general legal principle governs this type of clause?
What is the standard risk framework for evaluating it?

— Phase 2: application (feeds on Phase 1 output)
Now apply those principles to the following clause
and identify the specific risks for a SaaS vendor:

{{clause text}}

Use when: the task requires applying general domain knowledge to specific cases — legal analysis, medical diagnosis reasoning, technical root-cause analysis. Step-back outperforms standard CoT on tasks where the model needs to retrieve deep domain knowledge before reasoning about the specific instance.

Plan-and-solve prompting

Wang et al. (2023) identified a specific failure mode of zero-shot CoT: the model sometimes generates plausible-sounding reasoning steps in the wrong order, or skips steps it should not.^#PlanSolve Plan-and-solve addresses this by splitting the CoT trigger into two explicit phases: plan first, then execute the plan.

Let's first understand the problem and devise a plan
to solve it. Then, let's carry out the plan step by step
and solve the problem.

Use when: the task involves multi-step planning or sequential decisions — writing a strategy document, designing a system, debugging a pipeline. Plan-and-solve reduces the rate of step-order errors compared to unstructured CoT.

Tabular CoT

Jin et al. (2023) introduced Tab-CoT, which structures the reasoning trace as a table rather than free-form prose.^#Tabular Each row represents one reasoning step, with columns for the step content, the intermediate conclusion, and a confidence or verification flag. The structured format prevents the model from generating verbose, meandering reasoning chains that lose track of intermediate results.

Reason through the problem using a table with three columns:
| Step | Reasoning | Intermediate result |

Complete each row before moving to the next.
After the table, state your final answer in one sentence.

Use when: reasoning chains tend to become long and lose coherence, or when the task involves tracking multiple variables simultaneously. Tabular CoT is especially effective for financial analysis, multi-variable optimisation, and eligibility or compliance reasoning.

Analogical reasoning

Yasunaga et al. (2023) showed that prompting the model to first identify and describe an analogous problem it has seen before — and then map the reasoning from that analogous case onto the current problem — improves performance on novel tasks where no direct few-shot examples are available.^#Analogical

Before solving this problem:
1. Identify a simpler or more familiar problem that has
   the same underlying structure.
2. Describe how you would solve that analogous problem.
3. Map that solution approach onto the current problem
   and solve it.

Use when: the task is genuinely novel or atypical, no domain-specific few-shot examples are available, and you need the model to leverage general problem-solving patterns rather than memorised templates.

Contrastive CoT

Chia et al. (2023) demonstrated that including both a correct reasoning trace and an explicitly incorrect reasoning trace in the few-shot examples — with the incorrect one clearly labelled — further improves accuracy compared to correct-only examples.^#Contrastive Seeing where reasoning goes wrong, and how, helps the model learn the decision boundaries of the task more precisely.

Question: Is 17 a prime number?

✗ Incorrect reasoning (labelled)
17 is odd. Odd numbers are prime. Therefore 17 is prime.
Error: the premise "odd numbers are prime" is false (9, 15, 25…)

✓ Correct reasoning (labelled)
17 is only divisible by 1 and itself.
Check: 17 ÷ 2 = 8.5, 17 ÷ 3 = 5.67, 17 ÷ 4 = 4.25 — none divide evenly.
No factor exists between 1 and √17 ≈ 4.12. Therefore 17 is prime. ✓

Use when: the task has common failure modes or systematic error patterns — classification tasks with easily confused categories, logical inference with a common fallacy, data extraction where a specific field is routinely misidentified. Contrastive CoT directly targets these failure modes at the example level.

4. Self-consistency — sampling multiple reasoning paths

Wang et al. (2022) identified a fundamental limitation of greedy CoT: when the model generates a single reasoning chain, the quality of the final answer depends entirely on whether that particular chain happened to be correct.^#Majority Errors early in the reasoning trace compound; the model has no mechanism to detect that it has gone wrong.

Self-consistency addresses this by sampling multiple independent reasoning chains from the same prompt — at a non-zero temperature — and taking the majority answer across chains as the final output. The intuition: if ten different reasoning paths all arrive at the same answer, that answer is more likely to be correct than any single path alone. Different chains make different errors; correct paths share a common destination.

Wang et al. showed improvements of 17.9 percentage points on GSM8K (grade-school math), 11.0 points on SVAMP (arithmetic word problems), and 12.2 points on AQuA (algebra) compared to standard CoT, using PaLM 540B.^#Majority The improvement is most pronounced on tasks with a single correct answer that multiple valid reasoning paths can reach.

The cost trade-off

Self-consistency multiplies your API cost and latency by N. This is not a technique for every call — it is a targeted investment for high-stakes decisions where reliability justifies the cost. The canonical use cases are:

• High-consequence classification (medical triage, legal risk assessment, fraud detection)
• Multi-step quantitative reasoning where a single error would be costly
• Automated decision systems where human review is not available downstream
• Evaluation of other prompts — using self-consistency as the gold standard against which simpler prompts are compared

5. Tree of Thoughts — extending CoT to deliberate search

Both standard CoT and self-consistency are linear in time: they generate a sequence of tokens from start to finish. Yao et al. (2023) introduced Tree of Thoughts (ToT) as a fundamentally different architecture: a search over a tree of possible reasoning states, with the model acting as both generator and evaluator at each node.^#Tree

In ToT, the model generates multiple candidate "thoughts" (partial reasoning steps) at each branch point. A separate evaluation step — either another model call or the same model prompted as an evaluator — scores each candidate. Only the highest-scoring branches are expanded further. The result is a deliberate, tree-structured search over the reasoning space, rather than a single left-to-right pass.

Yao et al. demonstrated ToT on the Game of 24 (find an arithmetic expression using four numbers that equals 24), creative writing with structural constraints, and a crossword mini-puzzle benchmark. Standard GPT-4 with CoT solved 4% of Game of 24 problems; ToT with GPT-4 solved 74%.^#Tree

ToT in production: the real constraints

The performance gains are real. The cost is also real: ToT requires N × K × depth model calls per task, where N is the number of nodes evaluated, K is the branching factor, and depth is the tree depth. A modest ToT configuration (depth 3, K = 3, 3 evaluation calls per node) requires approximately 30–40 model calls per task. This places ToT firmly in the category of offline reasoning tasks — batch processing, research pipelines, long-horizon planning — not real-time user-facing applications.

The practical deployment pattern is to use ToT for generating high-quality outputs offline that are then cached, stored, or used as few-shot examples for lighter online models. Treat ToT as an expensive generation process whose outputs are assets, not as a runtime inference strategy.

6. Reasoning models change the equation

The release of OpenAI's o1 in September 2024, followed by o3, o4-mini, and Anthropic's Claude Sonnet 4.6 / Opus 4.6 with extended thinking, introduced a new model category that makes one critical change to the CoT picture: the reasoning chain is generated internally, before the model's visible output begins.

These models perform extended chain-of-thought reasoning in a hidden scratchpad — a sequence of tokens that is generated but not returned in the standard API response. The model reasons through the problem, backtracks when needed, explores alternative approaches, and only then produces the final response. The visible output is the conclusion of a reasoning process the user does not see.

Anthropic's documentation for Claude Sonnet 4.6 / Opus 4.6 explicitly notes that prompting for explicit step-by-step reasoning in the user turn is unnecessary and may produce worse results than allowing the model's extended thinking to run uninterrupted.^#Extended OpenAI's guidance for o1, o3, and o4-mini models makes the same recommendation: avoid CoT-style instructions in the prompt; focus on clear task specification.

Self-consistency's value also diminishes on reasoning models, because the internal search already explores multiple reasoning paths within a single generation. Sampling N responses still provides some gain, but the marginal return per additional sample is lower than on standard models. The cost-to-benefit trade-off shifts accordingly.

7. When not to use chain-of-thought

CoT is often applied reflexively to any task that seems "hard." This is a mistake. Four conditions make CoT counterproductive.

Simple or single-step tasks

CoT adds tokens, latency, and cost. On tasks that require no reasoning chain — look up a fact, format a string, extract a named entity from a sentence — CoT produces verbose output with no accuracy benefit. The intermediate steps are padding, not reasoning. Use zero-shot with a tight format specification instead.

Underpowered models

On models below approximately 7B parameters (post-2024 training), CoT can reduce performance by generating plausible-looking but incorrect reasoning steps that condition the answer toward a wrong region. The model lacks the capacity to leverage intermediate steps as meaningful constraints. If you are working with a small, fast model, test CoT carefully against your baseline — do not assume it helps.

Latency-critical applications

Real-time autocomplete, speech transcription post-processing, interactive chatbots with sub-second response requirements — these cannot absorb the additional tokens that CoT reasoning chains require. For latency-critical paths, use direct prompting with a very tight format spec and accept the accuracy trade-off. Reserve CoT for batch processing or pre-generation pipelines where latency is not the constraint.

Reasoning models (as established above)

Prompting o1, o3, o4-mini, or Claude Sonnet 4.6 / Opus 4.6 with extended thinking to "think step by step" in the user turn interferes with the model's internal process. This is the single most common mistake practitioners make when migrating from standard to reasoning models. The visible output will look more verbose, but the quality is often lower because the user-turn CoT instruction competes with — rather than augments — the model's internal reasoning.

8. Decision guide — choosing the right CoT variant

The choice of CoT variant should be driven by the task type, the model, and the cost constraints. This table maps the most common production scenarios to the appropriate technique.

Lesson 04 moves from individual reasoning techniques to the architecture level: how to design and evaluate prompt chains, when to break a task across multiple model calls, and how to structure the handoffs between them so that errors in one step do not propagate silently through the pipeline.

References

1. #Wei Wei, J., et al. (2022). Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. The original CoT paper. NeurIPS 2022. arXiv:2201.11903
2. #ZeroShot Kojima, T., et al. (2022). Large Language Models are Zero-Shot Reasoners. Introduced "Let's think step by step" as a zero-shot CoT trigger. NeurIPS 2022. arXiv:2205.11916
3. #Survey Schulhoff, S., et al. (2024). The Prompt Report: A Systematic Survey of Prompting Techniques. Co-authored with OpenAI, Stanford, Microsoft, Princeton, Google, and 26 other institutions. Last updated Feb 2025. arXiv:2406.06608
4. #StepBack Zheng, H. S., et al. (2023). Take a Step Back: Evoking Reasoning via Abstraction in Large Language Models. Introduces step-back prompting; shows consistent improvements on STEM and knowledge-intensive tasks. ICLR 2024. arXiv:2310.06117
5. #PlanSolve Wang, L., et al. (2023). Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models. Identifies step-order errors in zero-shot CoT; introduces the plan-then-execute trigger. ACL 2023. arXiv:2305.04091
6. #Tabular Jin, Z., et al. (2023). Tab-CoT: Zero-shot Tabular Chain of Thought. Structures reasoning as a table; reduces verbosity and step-tracking errors. ACL Findings 2023. arXiv:2305.17812
7. #Analogical Yasunaga, M., et al. (2023). Large Language Models as Analogical Reasoners. Demonstrates self-generated analogies as a CoT augmentation. ICLR 2024. arXiv:2310.01714
8. #Contrastive Chia, Y. K., et al. (2023). Contrastive Chain-of-Thought Prompting. Shows that including labelled incorrect reasoning examples alongside correct ones further improves accuracy. arXiv:2311.09277
9. #Majority Wang, X., et al. (2022). Self-Consistency Improves Chain of Thought Reasoning in Language Models. Introduces majority-vote sampling over multiple CoT chains. ICLR 2023. arXiv:2203.11171
10. #Tree Yao, S., et al. (2023). Tree of Thoughts: Deliberate Problem Solving with Large Language Models. Introduces tree-structured search over reasoning states; 74% vs 4% on Game of 24 with GPT-4. NeurIPS 2023. arXiv:2305.10601
11. #Extended Anthropic. (2025). Extended Thinking — Claude API Documentation. Covers the extended thinking parameter, thinking budgets, and guidance on prompt design for reasoning models. platform.claude.com/docs/en/build-with-claude/extended-thinking