An agent is a loop that keeps asking a model what to do next. That was the previous piece. This one is about the strangest property of the thing sitting at the centre of that loop.
The model has no memory.
Not "a short memory." None. Every scrap of continuity you experience is something the surrounding software resent on your behalf.
What the model actually sees, and what it keeps afterwards
Worth being precise here: products built on top of models often do have memory. ChatGPT and similar chatbots maintain long-term memory as a layer above the model. But the model itself retains nothing between calls.
What one invocation actually contains
Take a single agent invocation. It contains exactly one system message, one user message (the prompt), zero or more tool messages, and one final response from the model.
The system message carries the general instructions: the agent's role and capabilities, initial knowledge, output specifications, examples. It is usually long.
One invocation is the whole unit. When it ends, the model retains nothing from it.
Long conversations are just replayed history
A long conversation is several invocations of the agent loop, each with its own final response, stitched together by the surrounding system feeding every prior message into the next one.
An example makes it concrete. A user asks whether routing routine support tickets to an agent would pay for itself. The agent responds with a clarifying question as its final response: should the comparison cover last quarter, or the whole year? The controlling system shows that to the user, who picks last quarter. Now a completely new invocation begins, carrying the full history of the first one plus the new choice. Only because that history was replayed does the agent "know" the answer is last quarter, and it can start reading the export and working through the arithmetic until it returns a final response.
The agent did not remember. The system re-told it.
The context window is the working memory
The context window is the total amount of information a model can process and reason over in a single invocation. Not only text: images, audio, structured data and embeddings too, depending on the model.
The window acts as the model's working memory, and it determines what the model can see when it generates a response.
One consequence is worth internalising: every high-level concept people talk about, agent memory, goals, state, must eventually be turned into tokens and fitted into the context window of every single API request.
When the model responds, it draws on knowledge encoded in its weights during training. But everything in the context window is forgotten the moment the response is generated. Providers may store it for auditing or future training, but the model does not learn or retain any of it. The only way to carry information across invocations is to store it at the framework or system level and feed it back in next time.
A larger context window buys real capability: longer conversations, richer inputs, more complete reasoning over prior steps, detailed references to complex artifacts. A smaller one constrains continuity and the ability to handle deeply nested tasks.
What to do when you run out of room
Even a large window fills up on long conversations or tasks with big inputs and outputs. Every model has its own limit, which is a real factor in choosing one.
The usual options are to truncate the history, summarise it, or compress it into some smaller form.
Multi-agent systems offer a different answer: divide and conquer. Each agent owns a specific sub-task and dedicates an entire context window to just that. Only the results, which are typically much smaller, get aggregated by a higher-level agent.
| Strategy | What it does | What it costs |
|---|---|---|
| Truncate | Drops the oldest messages | The dropped detail is gone for good |
| Summarise | Replaces history with a shorter account | Whatever the summary left out |
| Compress | Re-encodes the same content smaller | Fidelity, and the work of compressing |
| Divide and conquer | Gives each sub-task its own window | Coordination between the agents |
Why providers keep it stateless
The obvious question: why can't the provider just remember the conversation? In theory it would save enormous bandwidth. Send only the new message instead of the entire history every time.
The answer is a distributed-systems problem, not an AI one.
At scale you want many copies of a model running in parallel, handling a massive number of requests. If each instance held its own memory, you would need session affinity, pinning every user's requests to the instance holding their history.
Then there are the gaps. Users pause between messages. Provider-managed session memory has to survive those idle periods, and that adds up quickly on long-running sessions.
The volume is the next problem. Imagine 10,000 users sending 10 TB within five minutes. All of it has to be stored somewhere to serve the next message. In memory? A database? A distributed cache? Each choice carries different costs. Then every new message means fetching that user's context and injecting it, which introduces failure modes and latency.
Finally, termination. The provider must either keep session state forever so any conversation can be resumed, or invent heuristics for when to delete it. Neither is comfortable.
So providers generally prefer to keep the model stateless and let frameworks manage session state. That improves scalability, reduces latency, and lowers cost.
The practical consequence
Once you internalise that the model is stateless, a lot of agent design stops being mysterious.
Context is a budget, and you are spending it on every call. Memory is a feature you build, not one you receive. And the reason your agent "forgot" something is almost never the model. It is that your system did not put it back in the window.