Context windows, memory, and multi-turn chat

Welcome back. This is the fourth walk in our series on large language models. Last time we drew a clear line between training, where the model's weights were set, and inference, the live call where you send a prompt and get a completion. Today we zoom in on that live call and ask a deceptively simple question. What can the model actually see when you are chatting with it, and why does it sometimes seem to forget what you just told it?

Let us start with a mental image. Imagine you are sitting in a room with a very capable colleague. This colleague has read an enormous amount of material and can reason about language fluently. But there is a catch. Every time you ask them something, they walk out of the room, close the door, and come back in fresh. They have no memory of having spoken to you before. The only thing they know about your conversation is what you hand them on a sheet of paper each time you knock on the door. That sheet of paper is the context window.

This is the single most important thing to internalise about chat interfaces. When you use something like a chat product and you see a long thread of back and forth, it feels like the model is remembering the whole conversation. But under the hood, the model itself is not maintaining a memory across turns. The application around the model is doing that. Each time you press send, the application takes your entire conversation history so far, packages it up, and sends it as one big input to the model. The model reads the whole thing fresh and produces the next response. It is not continuing a thought. It is re-reading everything and responding anew.

So the effective memory of the model is not some internal brain state. It is the context window, the text you send on each call. If it is in the window, the model can see it. If it is not in the window, the model cannot see it. That is the whole rule.

Now, context windows have a size limit. Think of it as the size of that sheet of paper. It is measured in tokens, which are the chunks of text the model processes. Every model has a maximum number of tokens it can accept as input on a single call. Some models have relatively small windows, others have very large ones, but none are infinite. And here is where things get interesting in a multi-turn chat.

Imagine you have been chatting for a while. You sent a first message, the model replied, you sent another, it replied again, and so on. Each of those turns takes up tokens. The application is accumulating all of them and resending the growing history each time. At some point, the total gets close to the window limit. What happens? The application has to make a decision. Typically, it starts dropping the oldest turns from the beginning of the conversation. Your earliest messages and the model's earliest replies simply fall off the sheet of paper. The model no longer sees them because they are no longer being sent.

This is why a model can seem to forget something you said twenty minutes ago. It is not that the model has a faulty memory in the way a person might forget. The model never had a persistent memory at all. The application was re-sending the history, and when the history got too long, the oldest parts were trimmed. From the model's perspective, those early turns never existed on this call. It is reading a truncated version of the conversation.

Let me give you a concrete example. Suppose you start a chat by saying you are a senior engineer working on a payments system in Python, and you want help refactoring a transaction module. The model acknowledges that context. You go back and forth for a long time, discussing various approaches. Eventually the conversation has grown long enough that the early turns get dropped. Now you ask a question that depends on the model remembering you are working on a payments system in Python. But that original message is gone from the window. The model might still infer Python from the code snippets that remain in more recent turns, or it might not. It might start giving you advice in a different language or lose track of the domain entirely. You would experience this as the model forgetting, but mechanically what happened is that the context was trimmed.

It gets more complex when you realise that the context window is not just holding your conversation turns. In a well-built application, there are other things competing for that space. There is often a system instruction at the very top, sometimes called a system message or system prompt. This is the behind-the-scenes text that tells the model how to behave. It might say something like you are a helpful coding assistant, always explain your reasoning, and use British English spelling. That takes up tokens. Then there are your user messages and the model's responses. And if the application uses tools, like fetching a web page or running a code snippet, the results of those tool calls also get inserted into the context. All of these things sit in the same window and compete for the same limited space.

So picture the window as a single pipe with a fixed diameter. System instructions flow in first. Then your conversation history. Then any tool results. If the pipe is full, something has to give. Usually it is the oldest conversation turns that get cut, because the system instructions are typically pinned at the top and kept. But the point is, everything is sharing one finite space.

Now, what do you do when you have a genuinely long conversation or a large amount of material and you cannot fit it all? Two main strategies have emerged in practice, and you should know both.

The first is summarisation. Instead of sending the entire raw history, the application or you periodically compress earlier turns into a summary. You take the first thirty minutes of back and forth and replace it with a paragraph that captures the key decisions and facts. That summary takes up far fewer tokens than the original turns. The model reads the summary plus the recent turns and has a reasonable approximation of the full history. You lose detail, but you gain longevity. Some applications do this automatically. You can also do it yourself by asking the model to summarise the conversation so far and then starting a fresh chat with that summary pasted in.

The second strategy is retrieval. Instead of putting everything into the context, you store material outside the model in a database or a search index. When you ask a question, the application searches that store for the most relevant pieces, pulls them in, and inserts just those pieces into the context window alongside your question. The model never sees the full document set. It sees a focused subset. This is the foundation of what people call retrieval-augmented generation, and it is how many enterprise applications handle large knowledge bases. The model's context window becomes a working desk where the most relevant pages are placed, rather than a filing cabinet holding everything.

Both strategies acknowledge the same fundamental constraint. The window is finite, so you must be deliberate about what goes in it.

Let me now address something that confuses many people. You have a chat with a model on your phone on Tuesday. You come back on Wednesday, open the same app, and the model seems to know what you talked about. Does the model remember you? No. The model is the same frozen weights it was before. What happened is that the application stored your conversation history in its own database, tied to your account. When you came back, the application loaded that history and sent it to the model as context. The memory lives in the application, not in the model.

This matters because it tells you something about stateless servers. The model serving infrastructure typically does not keep state between calls. Each call is independent. The model does not secretly remember you between sessions. If the application did not store and resend the history, the model would start every session completely fresh, with no knowledge of prior conversations. Some products give you the impression of continuity, but that continuity is engineered by the application layer, not by the model itself.

This also means that if you use an application programming interface directly, calling the model from your own code, you are responsible for managing the context. The model will not remember anything from your previous call unless you send the relevant history again. You are the application. You decide what goes in the window.

So let us turn this into practical habits you can carry forward.

First, keep your current goal in the prompt. Do not assume the model will infer your objective from something said many turns ago, especially in a long conversation. If the goal matters for this response, state it or restate it.

Second, refresh key facts when they matter. If it is important that you are working in Python, or that your audience is non-technical, or that the deadline is Friday, put that in the prompt again when it becomes relevant. Do not rely on the model having seen it forty turns ago.

Third, do not assume infinite memory. If a conversation is getting long and the model starts drifting or losing track, consider that the early context may have been trimmed. Start a fresh conversation with a concise summary of what matters, and continue from there.

Fourth, if you are building something yourself, be deliberate about what you send. Put the stable instructions in the system message. Put the conversation history in the middle. Put the immediate question or task at the end. Think about the window as a budget and spend it on what matters for this specific response.

Fifth, when dealing with large documents or knowledge bases, do not try to stuff everything into the context. Use retrieval to bring in only the relevant parts. The model performs better with a focused, well-chosen context than with a bloated one.

Let me give you one more image to seal this. The model is a brilliant reader with no notebook. Every time you ask it something, it reads whatever you place in front of it, responds, and then forgets everything. The application is the notebook. It writes down the conversation and shows the model the relevant pages each time. But the notebook has limited pages. When it fills up, the oldest pages are torn out. Understanding this separation between the model and the notebook is what separates people who are surprised by the model's apparent forgetfulness from people who design around it deliberately.

On the next walk, we will look at grounding, which is the practice of anchoring the model's response in specific, verifiable material you provide in the context. It is the natural complement to what we discussed today. If the context window is the model's working desk, grounding is about making sure the right documents are on that desk and that the model actually uses them rather than improvising. For now, let this settle. The model sees what you send, nothing more. Manage the window, and you manage the model's effective memory.

Quick recap. The model has no persistent memory between calls. Its effective memory is the context window, the full text you send each time. In a chat product, the application packages your conversation history and resends it on every turn, which creates the illusion of continuous memory.

Context windows are finite, measured in tokens. When the window fills, the oldest turns are typically dropped. This is why a model can seem to forget something you said earlier. It is not forgetting. The text simply is not being sent anymore.

System instructions, your messages, the model's replies, and tool results all share the same window. They compete for limited space. When the conversation is too long, you can use summarisation to compress earlier turns, or retrieval to pull in only the relevant material rather than sending everything.

The model itself does not remember you between sessions. Any continuity you experience comes from the application storing and resending your history. The serving infrastructure is stateless by design.

Practical habits: keep your goal in the prompt, refresh key facts when they matter, do not assume infinite memory, and if you are building something, treat the context window as a budget. Spend it on what matters for the response you need right now.