Fine-tuning and LoRA

What supervised fine-tuning minimizes

Given a model with weights $\theta$ and examples $(x_i, y_i)$ (an input and the desired answer), supervised fine-tuning keeps minimizing the cross-entropy,

with the same gradient descent from lesson 4 and the backpropagation gradients from lesson 8. In a language model, $y_i$ is a sequence of tokens and the sum runs only over the answer tokens. It starts from the pretrained weights $\theta_0$ (lesson 23), with much smaller learning rates than in pretraining and only a few passes over the data. LIMA (Zhou et al., 2023) used 1,000 examples, about 750,000 tokens, on the 65-billion-parameter LLaMA, which was pretrained on 1.4 trillion tokens (Touvron et al., 2023).

LoRA: the math

For a pretrained weight matrix $W_0\in\mathbb{R}^{d\times k}$, LoRA freezes it and writes the update as

The forward pass is $h = W_0x + \tfrac{\alpha}{r}B(Ax)$. $A$ starts as a Gaussian and $B$ as zeros, so $\Delta W = 0$ at the start (Hu et al., 2021); only $A$ and $B$ receive gradients. Since $\partial L/\partial A$ is proportional to $B$, only $B$ moves on the first step.

Each column $b_j$ of $B$ and each row $a_j$ of $A$ form a rank-one piece, $b_j a_j$ (column times row, like the lesson’s multiplication table): $BA = \sum_{j=1}^{r} b_j a_j$. Hence the name: the correction has rank at most $r$.

Trainable parameters: $r\,(d+k)$ instead of $dk$. It saves only if $r < \dfrac{dk}{d+k}$; for a square matrix, $r<d/2$. With $d=k=32$ (the tiny model’s middle layer), $r=16$ is a tie: $16\cdot 64 = 1{,}024$.

Once trained it can be merged, $W = W_0 + \tfrac{\alpha}{r}BA$, and inference costs the same as the original model; or the adapter can stay separate to swap it for another.

For GPT-3 175B ($d_{\text{model}}=12{,}288$, 96 layers), adapting only $W_q$ and $W_v$ with $r=4$ gives $96\cdot 2\cdot 4\cdot(12{,}288+12{,}288)=18{,}874{,}368$ trainable parameters: about 9,300 times fewer than 175 billion (the paper says “roughly 10,000×”). In FP16 the adapter takes about 35 MB versus the model’s 350 GB, and 100 adaptations take about 354 GB instead of 35 TB (Hu et al., 2021).

Why the change has low rank

If $M=\sum_i \sigma_i u_i v_i^\top$ is the singular value decomposition, the Eckart–Young theorem says the best rank-$r$ matrix (in Frobenius norm) is the sum of the first $r$ terms, and its error is $\sum_{i>r}\sigma_i^2$. In the tiny model, the change that full fine-tuning leaves in the 32×32 layer has singular values that fall quickly: the first four gather about 97% of $\sum\sigma_i^2$. That is what LoRA assumes and exploits; Aghajanyan et al. (2020) had already measured that fine-tuning a language model needs very few effective dimensions. It is not a law: Hu et al. do not expect a small rank to work for every task (for example, a language different from the pretraining one), and Biderman et al. (2024) measured that, in programming and math, full fine-tuning learns changes whose rank is 10 to 100 times higher than typical LoRA configurations.

Forgetting, replay, and what the evidence says

With narrow data, moving shared weights shifts whatever that data does not cover. In the tiny model (70 new examples from a single area, 300 full-batch Adam steps, one run with fixed seeds):

tuning	trainable	accuracy on the new part	accuracy on what it knew
none (base model)	—	0%	100%
full	1,251	98%	93%
full + 30 for replay	1,251	94%	97%
LoRA r = 2	128	91%	87%
LoRA r = 2 + 30 for replay	128	93%	98%

Here LoRA without replay forgets more than full fine-tuning; in large models, Biderman et al. (2024) found the opposite on average (LoRA learns less and forgets less), but not that it stops forgetting. Moral: forgetting is a risk with both methods, and you measure it with data the tuning did not see. The name “catastrophic” comes from McCloskey and Cohen (1989); in the toy the loss is moderate.

Memory: from sixteen bytes per parameter to less

Training with Adam in mixed precision stores, per parameter: FP16 weights (2 bytes), FP16 gradients (2) and, in FP32, a copy of the weights, momentum, and variance ($4+4+4$): $16$ bytes (Rajbhandari et al., 2019; lesson 25). With LoRA, frozen weights only need their copy (2 bytes in FP16) and the 16 bytes are paid only for $A$ and $B$. QLoRA also stores the base in 4 bits (NF4 format, with “double quantization” of the constants) and trains the adapter in 16 bits. Our own count for a 7-billion-parameter model, without activations: full fine-tuning $7\cdot 16=112$ GB; FP16 base, 14 GB; 4-bit base, about 3.5 GB, plus the adapter and activations.

The papers’ figures do not come exactly out of this count, because they depend on what is stored and how: the LoRA paper gives, for GPT-3 175B, from 1.2 TB to 350 GB, and the QLoRA paper counts more than 780 GB to fine-tune 65 billion parameters in 16 bits (about 12 bytes per parameter). Always compare under the same count.

The “Q” part can be tried in the tiny model: with its three matrices in 4 bits (NF4, blocks of 64), the map changes in 22 of 1,600 cells, and rank-2 LoRA with replay reaches 91% on the new part and 98% on what it knew, versus 93% and 98% with the 16-bit base. It is a toy: it reproduces the idea, it proves nothing about large models.

In code

import numpy as np

# 1) the count: one GPT-3 matrix (12288 x 12288) and rank 8
d = k = 12288
r = 8
full = d * k
lora = r * (d + k)
print(full, lora, full // lora)            # 150994944 196608 768

# 2) the correction starts at zero (B = 0): the model does not change
rng = np.random.default_rng(0)
W0 = rng.normal(size=(6, 6))               # frozen matrix (tiny)
A = rng.normal(size=(2, 6))                # r = 2, Gaussian
B = np.zeros((6, 2))                       # zeros
x = rng.normal(size=6)
h = W0 @ x + B @ (A @ x)                   # alpha/r = 1
print(np.allclose(h, W0 @ x))              # True

# 3) after training, B is no longer zero: the correction has rank <= 2
B = rng.normal(size=(6, 2)) * 0.1
print(np.linalg.matrix_rank(B @ A))        # 2

What comes next

Tuning on input-output pairs changes format, style, and task. Turning a model that only completes text into an assistant uses that same mechanism, but the question is which pairs and which signal: instruction tuning and reinforcement learning from human feedback (lesson 28). And it is practical: QLoRA was used to instruction-tune the Guanaco family (Dettmers et al., 2023), and those authors found that data quality matters far more than quantity (9 thousand OASST1 examples beat 450 thousand from FLAN v2 in their chatbot test).