Reasoning models

Chain of thought, in one formula

An autoregressive model generates each token conditioned on everything before it: $y_t \sim p_\theta(\cdot \mid x, y_{<t})$. If it answers directly, the answer $a$ comes from $p_\theta(a \mid x)$ in a few passes. With a chain of thought, it first writes $c = (c_1, \dots, c_m)$ and then answers conditioned on it:

Each token of $c$ costs a full pass, about $2N$ operations for an $N$-parameter model (lessons 24 and 26): with $N = 4\cdot 10^9$ and $m \approx 500$ that is about $4\cdot10^{12}$ more operations, and $m$ more decode turns. Wei et al. (2022) put it this way: the chain lets a problem be decomposed into steps, so that “additional computation can be allocated to problems that require more reasoning steps.”

With 8 worked examples in the prompt, on GSM8K (grade-school math problems): PaLM 540B went from 17.9 to 56.9%, PaLM 62B from 9.6 to 29.9%, and PaLM 8B dropped from 4.9 to 4.1% (Wei et al., table 2). The authors observed that smaller models produced “fluent but illogical” chains. Kojima et al. (2022) showed that adding “Let's think step by step” is enough: with text-davinci-002, GSM8K rose from 10.4 to 40.7%. In the scene, the 3 and 51 tokens were counted with the Qwen3.5 tokenizer, which splits every digit into its own token.

Verifiable rewards

In lesson 28, RLHF maximizes a reward model’s points $r_\phi$ without drifting too far from the starting model:

Reinforcement learning with verifiable rewards (RLVR, a name from Lambert et al., 2024) uses the same objective but swaps the judge for a function that checks: $r(x,y) = \alpha\cdot\mathbb{1}[\,\text{verify}(x,y)\,]$, for example “the final answer matches the correct one” or “the code passes the tests.” DeepSeek-R1-Zero (DeepSeek-AI, 2025) samples 16 answers per question and compares each reward with the group’s (GRPO): $A_i = (r_i - \text{mean}(r))/\text{std}(r)$. Its reward is accuracy + format; they avoided neural reward models because they are “susceptible to reward hacking.” On AIME 2024 its accuracy rose from 15.6 to 77.9% (one sample) and to 86.7% voting among 16. The average length of its responses grew during training, and the “aha moment” appeared (“Wait, wait. Wait.”). OpenAI describes o1 (2024) as “trained with large-scale reinforcement learning to reason using chain of thought”; it did not publish its recipe.

Self-consistency

With $k$ sampled answers $a_1, \dots, a_k$, pick the most repeated one:

Wang et al. (2023) voted among 40 chains: PaLM 540B went from 56.5 to 74.4% on GSM8K (their measurement of the single chain is 56.5%; Wei et al.’s, 56.9%). OpenAI reported that o1 solved 74% of AIME 2024 with one sample and 83% with consensus among 64 (GPT-4o: 12%). Voting helps when errors scatter; if the model is always wrong the same way, the vote confirms the error.

Our measurement

Qwen3.5-4B (Qwen, 2026; Apache-2.0), quantized to fp8 and run with vLLM on an RTX 3060. For each language, 32 complete chains of thought (up to 4,096 tokens) at temperature 0.7, top-p 0.95, and top-k 20. Forced budget, like the budget forcing of Muennighoff et al. (2025): if the chain finished on its own in $L \le B$ tokens, its answer is used; if not, it is cut at $B$ tokens, the thinking is closed with a fixed phrase, and the answer is requested. $B = 0$ is the model’s no-thinking mode. Checking the answer (is it 120?) is the same rule a verifiable reward would use.

budget (tokens)	0	32	64	128	256	512	1,024 – 4,096
right, question in Spanish	14	0	4	3	30	32	32
right, question in English	10	7	31	21	27	32	32

Complete chains ran between 470 and 628 tokens (mean 529) in Spanish and between 400 and 714 (mean 498) in English. With short budgets the result goes up and down; from 512 on it does not change. It is a single problem with 32 tries and a small model: it illustrates the phenomenon, it does not measure it in general.

In code

from collections import Counter
import random

correct = 120
no_thinking = [125, 165, 200, 125, 120, 150, 170, 120, 170, 120, 174, 125, 120, 123, 166, 145,
               135, 143, 164, 160, 181, 120, 202, 120, 143, 145, 120, 135, 120, 182, 120, 120]
cut_at_32 = [120, 125, 120, 125, 125, 125, 125, 120, 120, 125, 120, 125, 125, 125, 125, 125,
             125, 125, 125, 125, 125, 125, 120, 125, 120, 125, 125, 125, 125, 125, 125, 125]
length = [566, 532, 560, 477, 609, 486, 562, 531, 714, 567, 560, 417, 490, 475, 494, 401,
          493, 525, 501, 479, 457, 401, 473, 400, 468, 460, 484, 485, 474, 474, 466, 461]  # tokens each try thought

def hits(answers):
    return sum(a == correct for a in answers)

def vote(answers):                     # the most repeated answer and its votes
    return Counter(answers).most_common(1)[0]

print(hits(no_thinking), vote(no_thinking))
print(Counter(a for a in no_thinking if a != correct).most_common(1))   # the most repeated error
print(hits(cut_at_32), vote(cut_at_32))

# how often is the vote right if k tries, picked at random, vote?
rng = random.Random(0)
for k in (1, 5, 15):
    right = sum(vote(rng.choices(no_thinking, k=k))[0] == correct for _ in range(10_000))
    print(k, right / 10_000)

# thinking tokens with budget B (if the model finishes earlier, it stops on its own)
for B in (0, 256, 512, 4096):
    print(B, round(sum(min(l, B) for l in length) / len(length), 1))
# 10 (120, 10)
# [(125, 3)]
# 7 (125, 25)
# 1 0.3133
# 5 0.5068
# 15 0.8167
# 0 0.0
# 256 256.0
# 512 479.2
# 4096 498.2

(On a tie, most_common keeps the answer that appeared first.)

Cost and waiting

Anthropic, for example, bills thinking tokens as output, and bills the full thinking even when it shows only a summary. Voting among $k$ multiplies the generated tokens by $k$ (in our test, 32 tries with thinking add up to 16,915 tokens in Spanish and 15,942 in English). The scene’s 100 tokens per second is an assumption.

Limits

Wei et al. reviewed 50 chains that ended in a wrong answer: 46% were almost right, except for a minor error (a calculation, a symbol, or a missing step). Chen et al. (2025) gave models a hint hidden in the question and measured whether their thinking mentioned it when they used it: Claude 3.7 Sonnet did so in 25% of cases and DeepSeek-R1 in 39%. Written thinking helps with calculating and checking, but it is not a guaranteed faithful explanation.

What comes next

Every question in this lesson was independent. In a conversation, everything that came before also takes up the model’s window (and the thinking takes tokens too): that is lesson 30, context and memory.