How an agent remembers what you said

A chat with an AI feels continuous. You give your name, and ten messages later the assistant still uses it. The mechanism behind that feeling is one list that you keep on your own machine and send again in full on every call.

Everything below runs, and none of it needs a framework.

One call, in full

A single call to a model looks like this. Every action an agent takes is this call, repeated.

reply = client.chat(
    model=MODEL,
    messages=[
        {"role": "system", "content": "You are a support assistant. Be brief."},
        {"role": "user",   "content": "My name is Janatan."},
    ],
)

Two details in that code matter more than the rest.

The first detail is messages. The parameter takes a list, and the list holds the entire situation the model will see. Each call describes that situation from the beginning. The provider keeps no conversation id and no session handle on its side, so a call arrives complete or arrives without the context.

The second detail is role. Each item in the list carries a role label, and the label tells the model how to read that part: your standing instructions, the user speaking, or the model's own earlier reply. The label guides interpretation. Enforcement lives in your own code.

The reply that comes back carries three fields worth reading.

print(reply.content)        # "Nice to meet you, Janatan."
print(reply.stop_reason)    # "end_turn"   <- why it stopped
print(reply.usage)          # input_tokens=24, output_tokens=8

content holds the answer, and most code reads only this field.

stop_reason says why the model stopped talking. It finished the thought, or it reached your token limit, or it wants to call a tool. Code that reads this field can tell a complete answer from a truncated one, and a truncated answer looks exactly like a short answer.

usage counts the tokens the call consumed. That count drives the rest of this post.

Watch a model forget

Two separate calls show the mechanism directly. Read the second one closely.

first = client.chat(model=MODEL, messages=[
    {"role": "user", "content": "My name is Janatan."},
])

second = client.chat(model=MODEL, messages=[
    {"role": "user", "content": "What is my name?"},
])

print(second.content)
# "I do not have access to your name."

The second call carried one message, and that message contained only the question. The model answered from the only text it received. Your first call happened on your machine, in your variables, and its contents stayed there.

Fifteen lines that build a memory

You keep the list yourself and append to it as the conversation goes on.

history = [{"role": "system", "content": "You are a support assistant. Be brief."}]

def say(text: str) -> str:
    history.append({"role": "user", "content": text})      # what you said
    reply = client.chat(model=MODEL, messages=history)     # send EVERYTHING, every time
    history.append({"role": "assistant", "content": reply.content})   # what it said
    return reply.content

say("My name is Janatan.")
say("What is my name?")     # "Your name is Janatan."

The second call now works, and the list explains why. After two turns history holds four messages: your first message, the model's reply, your second message, and the reply to that. You send all four. The model reads the list from the top and answers from what it reads.

Print the list after those two turns and the whole mechanism is visible:

[
  {"role": "system",    "content": "You are a support assistant. Be brief."},
  {"role": "user",      "content": "My name is Janatan."},
  {"role": "assistant", "content": "Nice to meet you, Janatan."},
  {"role": "user",      "content": "What is my name?"},
]

The model's own earlier answer sits in that list as ordinary text, and you put it there. Each call is a fresh read of a document that keeps growing.

Nothing persists on the model's side. The list lives with you, and grows.

What it costs by turn ten

The cost of that growing list is plain arithmetic, and the result surprises most people.

Assume a 200-token system prompt, roughly 50 tokens per user message, and roughly 150 per reply. Those are ordinary numbers for a support conversation.

(Illustrative arithmetic from those stated assumptions, not a measurement.)

Two things fall out of that table.

Turn ten costs eight times turn one. The question stays the same length and the reply stays the same length. The history you re-send is the part that grew.

The cumulative total grows as a square. Ten turns of a short conversation costs 11,500 tokens. Ten lots of 250 would come to 2,500. Every message you add gets paid for again on every later turn, so an early message in a long conversation is billed dozens of times over.

A bigger context window raises the ceiling and leaves this arithmetic exactly where it is. The window measures the payload you send on every single call.

Three ways to keep the list small

The list belongs to you, so every technique for controlling it is code you write.

Trim it. Keep the system prompt and the last few turns, and drop the middle. Cheap to build, and the dropped detail is gone for good.

Summarise it. Replace ten old turns with one paragraph. Cheaper still, and you lose whatever the summary left out.

Leave things out from the start. Most agent history is tool output that mattered for one turn and stays dead weight afterwards. This option saves the most and gets skipped the most.

Each of those is a policy you write, in your code, about a list you own.

What to take from this

• An agent's memory is a list of messages held in your process and sent in full on every call.
• The cost of a conversation grows as a square, because every message is re-sent on every later turn.
• Keeping that list small is your job, and each way of doing it trades cost against detail.

The list explains how an agent holds a conversation. Getting a useful answer back is a separate problem: the reply arrives as a paragraph of English, and the code around it needs a number.