Long-term memory, what an agent keeps across sessions

A session carries a conversation from one turn to the next. Start a new conversation, with a new session, and the agent is a stranger again. Yesterday's "I work in marketing" is gone.

Long-term memory is what persists across sessions, and its defining trait is the opposite of a session's: it saves almost nothing, on purpose.

Two stores, two jobs

The SessionManager and the memory store solve different problems, and keeping them separate keeps each simple.

A session preserves continuity within one conversation. Call run() with the same session_id and the history is there; use a different session_id and you get a clean slate, by design. A session is not meant to leak between conversations.

Long-term memory is the deliberate bridge between conversations. It holds the handful of facts that should follow the user everywhere: who they are, what they prefer, what has been solved before. When a new session starts, the agent pulls relevant memories in, so it arrives already knowing what matters.

Selection is the whole design

Short-term memory captures everything automatically. If long-term memory did the same, it would be a transcript of every conversation the user ever had, which is useless to search and expensive to carry.

So long-term memory is selective. At the end of a run, an extraction step reads what happened and asks: what here is worth remembering next time? Most of a conversation is not. The few facts that are get saved; the rest is let go.

async def extract_memories(events: list[Event]) -> list[Memory]:
    """At the end of a run, pull out the few facts worth keeping."""
    transcript = render(events)
    return await ask_for(
        f"From this conversation, extract durable facts about the user or task "
        f"worth remembering in future sessions. Skip anything one-off.\n\n{transcript}",
        schema=list[Memory],
    )

The instruction carries the design. "Durable facts worth remembering", "skip anything one-off". A question about today's weather is one-off. "I prefer answers in Python" is durable. The extraction step is where that judgement gets made, once, at the end of the run.

Short-term keeps everything from one session. Long-term extracts the few durable facts that outlast it.

Simple facts are text, complex ones need structure

A one-line fact needs no ceremony. "The user is a marketer" is a sentence; store it and search it as text.

Some memories carry more than a sentence, and cramming them into free text loses the parts you will want to query later. A record of solving a hard problem is the clear case: you do not just want "solved it", you want what the problem was, what approach worked, and what the result was, each as its own field, so a future run can find and reuse the approach.

class TaskMemory(BaseModel):
    problem: str          # what was being solved
    approach: str         # the method that worked
    result: str           # how it turned out
    tags: list[str]       # for filtering later

Structured output, from earlier in this series, is exactly the tool for this. The extraction step returns memories shaped to a schema, so the fields you will search on are real fields, not phrases buried in a paragraph. A memory you can query by approach is worth far more later than a sentence you have to re-read.

The shape of the whole thing

Long-term memory has a clear lifecycle, and this post covers the front half.

at the end of a run:
  read the events -> extract the durable facts -> shape them (text or schema) -> store them

at the start of a later run:
  take the request -> find relevant memories -> add them to the context   (next post)

Extraction and shaping are the front half: deciding what to keep and in what form. Storing and retrieving, over a vector store, are the back half, and they reuse the retrieval machinery this series already built. The next post finishes the loop.

The front half of long-term memory: read the run, extract durable facts, shape them, store them.

What to take from this

• A session remembers within one conversation by design and stops there. Long-term memory is the deliberate bridge across conversations.
• Long-term memory is selective. An extraction step at the end of a run keeps the few durable facts and drops the one-off rest, which is what keeps it useful.
• Simple facts store as text; complex ones use structured output, so the fields you will search on later are real fields rather than phrases in a paragraph.

The facts are extracted and shaped. The back half is storing them where the agent can find the relevant ones later, which is the vector search this series already built, now aimed at the agent's own past. The next post assembles it.