Inference: how a model answers

Two phases, two bottlenecks

A dense model with $N$ parameters does about $2N$ operations for every token it processes (one multiplication and one addition per parameter; Kaplan et al., 2020; Pope et al., 2022), plus an attention term that grows with context.

Prefill (the message’s $L$ tokens at once; $n_\ell$ blocks, $d_{\text{at}}$ = heads × head dimension):

These are large matrix multiplications: the GPU works close to its peak. The lesson assumes 40% of peak (Llama 3 reports 38 to 43% in training; Pope et al. reach 76% in prefill with large batches). With the 8B model and 2,000 tokens: $3.3\cdot10^{13}$ FLOPs ÷ $(0.4\cdot 989\cdot10^{12})$ ≈ 84 ms.

Decode ($B$ conversations of $L$ context tokens each). Every turn has a memory time and a compute time (Pope et al., §2), and the larger one rules:

with $W$ the bytes of weights, $c_{kv}$ the cache bytes per token, BW the bandwidth, and $P$ the useful compute (the lesson uses half the peak). With $B=1$ there are $2N$ operations for every $W=bN$ bytes, that is $2/b$ operations per byte: one, in FP16. The H100 can do $989\cdot10^{12}/3.35\cdot10^{12}\approx 295$ per byte, so one person’s decode uses about 0.3% of the compute. Pope et al. put it this way: at small batch sizes and sequence lengths, loading the weights dominates; at large batches and long sequences, loading the KV cache does.

The KV cache, inside

The 2 is key and value; $n_{kv}$ the number of key-value heads; $d_h$ the head dimension and $b$ the bytes per number. Llama 3 8B (Llama Team, 2024, table 3): $2\cdot32\cdot8\cdot128\cdot2=131{,}072$ bytes per token (in the scene, $8\cdot128=1{,}024$ numbers per key per block). With full multi-head attention (32 KV heads) it would be $524{,}288$: Llama 3 uses grouped-query attention (GQA, Ainslie et al., 2023), which shares each pair among four query heads, “to reduce the size of key-value caches during decoding.” The same count gives Kwon et al.’s (2023) value for OPT-13B: $2\cdot5{,}120\cdot40\cdot2=819{,}200$ bytes, about 800 KB per token.

Without a cache, generating $G$ tokens after $P$ input tokens processes $GP+G(G-1)/2$ positions; with a cache, $P+G-1$. With $P=6$ and $G=10$: $105$ versus $15$. The toy transformer in the scene (2 blocks, 2 heads, $d=16$, random weights with a fixed seed; tiny.ts) runs both paths: it counts 105 and 15 passes, 498,240 and 72,960 multiply-adds, and its logits match with a maximum difference of 0.

Speed ceiling, batches, and cost

With the 8B in FP16 on an H100, 8,000 tokens of context, and 2 USD per GPU hour (assumption): $B=1$ gives 196 tokens/s and 2.84 USD per million; $B=60$, the most that fit, gives 42 tokens/s per person, 2,545 in total, and 0.22 USD per million: about thirteen times cheaper on the same machine. With 128,000 tokens of context only 3 conversations fit.

What an API user pays

On the lists consulted on October 12, 2026, output costs five times the input price on every Anthropic model, and five to more than eight times on Google’s. The scene’s prices ($3 and $15 per million) are an example with that ratio. If the start of your message repeats (instructions, a base document), prompt caching charges much less for it: on Anthropic’s list, reading from the cache costs 0.1 times the input price (0.05 or 0.025 on some models), and writing to it, valid for five minutes, 1.25 times.

The waits in the scene are estimates with the 8B model on one H100 and a single person: prefill at 40% of peak, and every writing turn with the memory time above.

Mixture of experts

Each block replaces its network with $E$ experts and a router that picks $k$ per token:

Mixtral 8x7B (Jiang et al., 2024, table 1): $d=4{,}096$, $h=14{,}336$, $n_\ell=32$, $E=8$, $k=2$. That gives 46.7 billion in total and 12.9 active. The paper puts it this way: compute is proportional to the active parameters, but “the memory costs for serving Mixtral are proportional to its sparse parameter count, 47B.” If each token picked its two experts at random (assumption: real routing is uneven), a batch of $B$ tokens would touch a fraction $1-(1-k/E)^B$ of the experts in each block: 25% at $B=1$, 90% at $B=8$, 99% at $B=16$.

In code

Cache, speed ceiling, and cost with the Llama 3 8B architecture:

layers, kv_heads, head_dim, bytes_num = 32, 8, 128, 2
kv_per_token = 2 * layers * kv_heads * head_dim * bytes_num         # 2 = key and value
print(kv_per_token)                                                  # bytes per token

weights = 8.03e9 * 2                                                 # FP16: 2 bytes per parameter
print(round(kv_per_token * 128_000 / 1e9, 2))                        # GB of one 128,000-token conversation

bandwidth = 3.35e12                                                  # H100 SXM: 3.35 TB/s
print(round(bandwidth / weights, 1))                                 # tokens/s ceiling for one person

def serve(people, context, usd_per_hour=2.0):
    bytes_per_turn = weights + people * context * kv_per_token       # weights + one cache per person
    t = bytes_per_turn / bandwidth                                   # seconds per turn
    throughput = people / t                                          # tokens/s of the whole system
    return round(1 / t, 1), round(throughput), round(usd_per_hour / (3600 * throughput) * 1e6, 3)

print(serve(1, 8000))      # (tokens/s per person, total tokens/s, US$ per million)
print(serve(60, 8000))
# 131072
# 16.78
# 208.6
# (195.8, 196, 2.837)
# (42.4, 2545, 0.218)

The cache does not change the result: single-head attention with and without a cache.

import numpy as np

rng = np.random.default_rng(0)
d = 8
Wq, Wk, Wv = (rng.normal(size=(d, d)) for _ in range(3))
x = rng.normal(size=(6, d))                          # 6 tokens (already turned into vectors)

def attention_no_cache(x):                           # recomputes K and V for every token
    q, k, v = x @ Wq, x @ Wk, x @ Wv
    s = q @ k.T / np.sqrt(d)
    s = np.where(np.tril(np.ones_like(s)) == 1, s, -np.inf)       # causal mask
    w = np.exp(s - s.max(1, keepdims=True)); w /= w.sum(1, keepdims=True)
    return w @ v

K, V, out = [], [], []                               # with a cache: one token per turn
for t in range(len(x)):
    q = x[t] @ Wq
    K.append(x[t] @ Wk); V.append(x[t] @ Wv)         # only the new pair enters the cache
    s = np.array(K) @ q / np.sqrt(d)
    w = np.exp(s - s.max()); w /= w.sum()
    out.append(w @ np.array(V))

print(np.allclose(attention_no_cache(x), np.array(out)))
# True

What comes next

Real systems add more pieces: PagedAttention splits the cache into blocks so memory is not wasted (vLLM: 2 to 4 times the throughput at the same latency as earlier systems, Kwon et al., 2023); continuous batching adds and removes conversations from the batch at every turn (Orca, Yu et al., 2022); and speculative decoding gave 2 to 3 times the speed on T5-XXL “with identical outputs” (Leviathan et al., 2023) and 2 to 2.5 times on Chinchilla 70B (Chen et al., 2023). All of this serves a base model; turning it into an assistant starts with fine-tuning, in lesson 27.