Context and memory

Stateless: what is really sent

In a chat API, the call for each turn carries the full list of messages; the model returns only the new message, and the app appends it to its list before the next turn:

{
  "system": "You are a logistics assistant. ...",
  "messages": [
    {"role": "user", "content": "My name is Ana and I work in logistics."},
    {"role": "assistant", "content": "Nice to meet you, Ana. How can I help?"},
    {"role": "user", "content": "What is my name?"}
  ]
}

Anthropic's documentation puts it this way: “The Messages API is stateless, which means that you always send the full conversational history to the API.” The system instructions go in a separate field, system. Some platforms store the history on their servers and reinsert it for you: the model is still stateless; only who stores the history changes.

What counts in the window

Per that same documentation, the window is “all the text a language model can reference when generating a response, including the response itself”: the system prompt, every message (with documents and images), tool definitions, and what the model writes on the turn. If the input alone already exceeds the window, the API returns an error (“prompt is too long”); what to do then is up to the app. The documentation notes that chat interfaces such as claude.ai can manage the window on a rolling “first in, first out” basis, or summarize the old part (“compaction”).

Why is there a limit? GPT-1 used “learned position embeddings” instead of the original paper's waves (lesson 22), and GPT-3 keeps that architecture with $n_{\text{ctx}}=2048$ (Brown et al., 2020): it learned one list per place and has no lists for more. Current models encode position differently, but the usable length still depends on the lengths they were trained on, and serving long contexts costs memory: the KV cache grows with every token (lesson 26). In Llama 3 8B (32 blocks, 8 key-value heads of 128 numbers, 2 bytes per number) that is $2\cdot 32\cdot 8\cdot 128\cdot 2 = 131{,}072$ bytes per token: at 128,000 tokens, almost 17 GB.

Model	Year	Window (tokens)	Source
GPT-1	2018	512	Radford et al.
GPT-2	2019	1,024	Radford et al.
GPT-3	2020	2,048	Brown et al.
Claude	2023	100,000 (≈ 75,000 words)	Anthropic
Llama 3.1	2024	128,000	Llama Team
Gemini 1.5	2024	over 1 million (up to 10 million in its tests)	Gemini Team

Fixed weights, examples in the context

The model can follow a pattern you show it inside the context (in-context learning): Brown et al. (2020) evaluate it “without any gradient updates or fine-tuning.” It is temporary: when those examples leave the window, the behavior disappears. Changing the weights takes training (lessons 4, 8, and 27). The GPT-4 technical report sums it up: it “has a limited context window, and does not learn from experience.”

What resending the conversation costs

Let $s$ be the size of the system prompt and $m$ the tokens each turn adds (your message $u$ plus the answer $a$). On turn $k$ the model processes $s + k\,m$ tokens (it reads everything before and writes what is new); over $n$ turns,

With $s=0$ and $m=100$: 5,500 tokens in 10 turns and 21,000 in 20, against 1,000 and 2,000 of text. The $m\,n^2/2$ term dominates: the total grows with the square of the turns. The instructions are paid for on every turn too: with $s = 2{,}000$, $u = 50$, $a = 150$, and 20 turns, the input adds up to $\sum_k \big(s + (k-1)m + u\big) = 79{,}000$ tokens, and half of them are the repeated instructions.

That is why prefix caching exists: the provider keeps the work already done on the repeated beginning (the KV cache from lesson 26) and charges less to reread it. In Anthropic's documentation, reading from the cache costs $0.1\times$ the base input price (less on some models) and writing to it costs $1.25\times$ (5-minute cache). If it hits on every turn, the previous example drops to about 14,628 equivalent tokens, 19%. Caching changes what you pay, not how much of the window is used.

Position matters: “Lost in the Middle”

Liu et al. (2023) gave each model 20 documents (about 4,000 tokens), only one with the answer, and changed its place. Accuracy (%), values from table 6 (appendix G); the last two columns come from table 1:

Model	1st	5th	10th	15th	20th	No documents	Only the right one
GPT-3.5-Turbo	75.8	57.2	53.8	55.4	63.2	56.1	88.3
Claude-1.3	59.9	55.9	56.8	57.2	60.1	48.3	76.1
LongChat-13B (16K)	68.6	57.4	55.3	52.5	55.0	35.0	83.4
MPT-30B-Instruct	53.7	51.8	52.2	52.7	56.3	31.5	81.9

These are 2023 models and a single task: the exact shape changes with the model and the test, and the authors find that a larger window, by itself, does not use the context better. RULER (Hsieh et al., 2024) extends the finding: among models claiming 32,000 tokens or more, only half kept satisfactory performance at that length. Anthropic's documentation calls it context rot: as the number of tokens grows, accuracy and recall degrade.

Memory outside the model

“Memory” features keep text outside the window and put it back in: a summary that replaces the old turns (Anthropic's compaction), or note files the model reads and writes across conversations (Anthropic's memory tool; MemGPT, by Packer et al., 2023, draws on an operating system's memory hierarchy). What is stored can become wrong or outdated, and it is information that leaves your organization.

In code

def trim(messages, limit, count):
    """Keep the instructions and the most recent turns that fit: first in, first out."""
    system, rest = messages[0], messages[1:]
    used, inside = count(system), []
    for m in reversed(rest):                        # newest to oldest
        if used + count(m) > limit:
            break
        inside.append(m)
        used += count(m)
    return [system] + inside[::-1]

def processed(turns, s, m):
    """Tokens the model processes: on turn k, the instructions s plus k turns of m tokens."""
    return sum(s + k * m for k in range(1, turns + 1))

def input_with_cache(turns, s, u, a, read=0.1, write=1.25):
    """Equivalent input tokens with prefix caching (assumed: it hits on every turn)."""
    m = u + a
    total = write * (s + u)                         # turn 1: everything is written
    for k in range(2, turns + 1):
        prefix = s + (k - 2) * m + u                # what was sent on the previous turn
        total += read * prefix + write * m          # read the prefix, write what is new
    return total

count = lambda m: len(m["text"].split())           # approximation: one word ≈ one token
history = [
    {"role": "system",    "text": "You are a logistics assistant."},
    {"role": "user",      "text": "My name is Ana and I work in logistics."},
    {"role": "assistant", "text": "Nice to meet you, Ana. How can I help?"},
    {"role": "user",      "text": "Give me an idea to save on shipping."},
    {"role": "assistant", "text": "Combine small orders into one truck."},
    {"role": "user",      "text": "What is my name?"},
]
inside = trim(history, limit=25, count=count)
print([m["role"] for m in inside])
print("Is the name still in the window?", any("Ana" in m["text"] for m in inside))
print(processed(10, s=0, m=100), processed(20, s=0, m=100))
without = sum(2000 + (k - 1) * 200 + 50 for k in range(1, 21))
cached = input_with_cache(20, s=2000, u=50, a=150)
print(without, round(cached), f"{cached / without:.0%}")
# ['system', 'user', 'assistant', 'user']
# Is the name still in the window? False
# 5500 21000
# 79000 14628 19%

What comes next

In lesson 31 (RAG) the external store becomes a full system: find the relevant fragments of your documents and put them in the window. In lesson 32, agents use that same idea, with memory in files, for jobs that last longer than one window.