From base model to assistant

Step one: supervised fine-tuning (SFT)

With pairs (prompt $x$, ideal answer $y$), instruction tuning minimizes cross-entropy only over the answer tokens:

The prompt tokens contribute no error (a “loss mask”). It is the loss from lesson 19 with different data, and the fine-tuning from lesson 27. InstructGPT used about 13 thousand prompts with demonstrations written by about 40 contractors (Ouyang et al., 2022). Earlier, FLAN (Wei et al., 2021) had turned dozens of datasets into instructions; later, LIMA (Zhou et al., 2023) fine-tuned a 65-billion-parameter model on only a thousand carefully chosen examples, to argue that almost all knowledge arrives in pretraining and tuning mostly teaches the format.

A caveat: the line between “base” and “assistant” has become blurry. The Qwen2.5 technical report (2024) says its pretraining already uses synthetic data generated by instruction-tuned models; with that base model the contrast does not show. That is why the scene uses TinyLlama 1.1B base (the 3-trillion-token checkpoint) and TinyLlama-1.1B-Chat-v1.0, which according to its model card was fine-tuned on UltraChat (synthetic dialogues) and then aligned with DPO on UltraFeedback (64 thousand prompts with answers ranked by GPT-4, not by people). The outputs are real, greedy-decoded and unedited (build-outputs.py); it is a small model trained mostly on English, and its assistant is imperfect: the “two-line” email does not come out in two lines.

Step two: the reward model (Bradley–Terry)

If $y_w$ is the preferred answer and $y_l$ the rejected one for prompt $x$, the Bradley–Terry preference model assumes

and the reward model $r_\phi$ is trained by minimizing $-\mathbb{E}\,\log \sigma\big(r_\phi(x,y_w) - r_\phi(x,y_l)\big)$. Only differences matter: adding a constant to every score changes nothing. In InstructGPT, labelers ranked between 4 and 9 answers per prompt (over about 33 thousand prompts), which gives $\binom{K}{2}$ comparisons per prompt, and the reward model had 6 billion parameters with a scalar output. Labelers agreed with each other 72.6% of the time (±1.5). While collecting training data they were asked to put helpfulness above truthfulness and harmlessness; for the final evaluations, the other way around.

The idea of learning a reward from human comparisons comes from Christiano et al. (2017), in Atari games and simulated robots; Stiennon et al. (2020) applied it to text summaries, and InstructGPT to general instructions. Preferences do not always come from people: Constitutional AI (Bai et al., 2022) replaces human labels about what is harmful with a written list of principles (RLAIF). In the scene, $r(y)=\mathbf{w}\cdot\boldsymbol{\varphi}(y)$ with four hand-assigned traits, and $\mathbf{w}$ is fit by logistic regression with a small L2 penalty.

Step three: optimizing on a leash (RLHF)

The model $\pi_\theta$ is tuned to maximize the reward without straying from the starting model $\pi_{\text{ref}}$; the “leash” is the KL divergence penalty:

Its exact solution has a closed form:

The scene computes that exact distribution; the “pressure” is $1/\beta$. In practice it is approximated with reinforcement learning algorithms such as PPO. InstructGPT applied a per-token KL penalty relative to the tuned model and also mixed in pretraining gradients (PPO-ptx) to reduce performance drops on some public tasks.

Optimizing too much: Goodhart’s law

Because $r_\phi$ is an imperfect stand-in, optimizing it too hard hurts real quality. Gao, Schulman, and Hilton (2022) measured it with a “gold” model playing the role of people: with $d=\sqrt{D_{\mathrm{KL}}}$, the gold score follows $R_{\text{bon}}(d)=d(\alpha-\beta d)$ when picking the best of $n$ samples and $R_{\text{RL}}(d)=d(\alpha-\beta\log d)$ with reinforcement learning: it rises, peaks, and falls. In their setup, the KL penalty did not measurably improve that frontier (it might depend on hyperparameters): it mainly serves to decide how far to go.

In the toy, the traits are independent in the starting model ($q = 0.5,\ 0.3,\ 0.4,\ 0.2$), and the optimum keeps that independence: the probability of each trait becomes $\sigma(\operatorname{logit} q_j + w_j/\beta)$. The practice judge has assumed weights $\mathbf{w}=(2.2,\ 0.5,\ 0.3,\ 0.45)$ and “real quality” is another assumed rule, $\mathbf{g}=(3,\ -1.2,\ 0.3,\ -2.6)$, which rewards correctness and penalizes padding and flattery. Real quality improves up to $1/\beta \approx 1.25$ (+0.90) and then falls, down to −0.97 at pressure 8, while the judge’s points always rise (from 0 to +1.89).

DPO: the model is secretly a reward model

Solving the closed form for $r$ gives $r(x,y)=\beta\log\frac{\pi^*(y\mid x)}{\pi_{\text{ref}}(y\mid x)}+\beta\log Z(x)$. Since Bradley–Terry depends only on differences, $Z(x)$ cancels and the loss depends only on the model (Rafailov et al., 2023):

No sampling from the model is needed during tuning: you start from a fixed set of pairs. In the scene, the model family is $\pi_\delta\propto\pi_{\text{ref}}\,e^{\delta\cdot\varphi}$; the DPO loss is computed over the 16 trait combinations with their normalization, and with the same regularization the weights $\beta\,\delta$ coincide with the reward model’s: an exact case of the equivalence. In a real model it is an approximation, because the model cannot represent every distribution. With preferences ranked by GPT-4 instead of people, Zephyr (2023) produced a competent assistant without new human annotation; TinyLlama Chat followed that recipe.

What the labs publish: OpenAI says ChatGPT was trained with RLHF using InstructGPT’s methods. Llama 2 used RLHF with rejection sampling and PPO (Touvron et al., 2023); Llama 3 used supervised fine-tuning, rejection sampling, and DPO, and explains that more complex reinforcement algorithms were less stable and harder to scale (Llama Team, 2024). For other commercial assistants the full recipe is not published: do not assume it.

What the layer costs

The “under 2%” comes from the InstructGPT paper: training the 175-billion-parameter tuned model took 4.9 petaflops/s-days, and the final PPO-ptx model, 60, versus 3,640 for GPT-3 (60 / 3,640 ≈ 1.6%; with the 4.9, ≈ 1.8%). It does not include the 6-billion-parameter reward model, data collection, or earlier experiments. In the evaluations, people preferred the outputs of the 175-billion-parameter InstructGPT to those of GPT-3 in 85% of cases (±3), and those of the 1.3-billion-parameter one to those of the 175-billion-parameter GPT-3.

In code

import math

def sigmoid(x):
    return 1 / (1 + math.exp(-x))

# 1) Reward model (Bradley–Terry): points of the preferred and the rejected answer
r_pref, r_rej = 1.8, -1.1
p = sigmoid(r_pref - r_rej)                # probability that the preferred one wins
print(round(p, 3), round(-math.log(p), 3))

# 2) Optimum on a KL leash: pi*(y) ∝ pi_ref(y) · exp(r(y) / beta), with three possible answers
pi_ref = [0.6, 0.3, 0.1]
r = [0.0, 1.0, 3.0]
for beta in (10.0, 1.0, 0.25):
    w = [q * math.exp(ri / beta) for q, ri in zip(pi_ref, r)]
    z = sum(w)
    print(beta, [round(x / z, 3) for x in w])

# 3) DPO: the same loss, but with the model's log-probabilities instead of points
beta = 0.1
logp_w, logp_l = -12.0, -15.0              # log pi_theta of the preferred and the rejected answer
ref_w, ref_l = -12.5, -14.0                # log pi_ref
margin = beta * ((logp_w - ref_w) - (logp_l - ref_l))
print(round(-math.log(sigmoid(margin)), 3))

Expected output:

0.948 0.054
10.0 [0.563, 0.311, 0.127]
1.0 [0.175, 0.238, 0.587]
0.25 [0.0, 0.001, 0.999]
0.621

With a large $\beta$ (a short leash) the model barely moves; with a small $\beta$ almost all the probability piles onto the best-scored answer and it loses variety.

What comes next

All of this shapes behavior from preferences. The next lesson uses the same reinforcement machinery, but with rewards that can be verified (the answer to a math problem is right or wrong) to teach the model to reason before it answers.