Building long-term memory with a vector store

The previous post extracted the durable facts from a run and shaped them. This post stores them and gets them back later, and the machinery is one you already built.

Long-term memory is retrieval-augmented generation pointed inward. The corpus this time is the agent's own accumulated experience, rather than a wiki.

The store is a vector store

Memories are found by meaning, the same as any other retrieval. "Have I solved something like this before?" is a similarity search over past problem records; "what does this user prefer?" is a similarity search over saved facts. So the store is a vector store, exactly like the knowledge base from the RAG chapter.

import chromadb

class TaskMemoryManager:
    def __init__(self):
        self.db = chromadb.Client().create_collection("task_memory")

Everything from the vector-search posts applies unchanged: each memory is embedded, stored beside its text, and retrieved by cosine similarity to a query. The novelty lies in what fills the store, and how it gets there, rather than in the retrieval.

The manager: extract, dedup, store

The memory manager runs the pipeline that turns a finished run into stored memories. Three steps, and the middle one is the one people skip.

async def remember(self, events: list[Event]):
    memories = await self.extract(events)          # 1. pull durable facts (previous post)
    for memory in memories:
        if self.is_duplicate(memory):              # 2. skip what we already know
            continue
        self.db.add(                               # 3. embed and store
            documents=[memory.text],
            metadatas=[memory.metadata],
            ids=[new_id()],
        )

Extract is the front half from the last post. Store is a vector-store write. Dedup is the step that keeps long-term memory from rotting.

Without dedup, every run that learns "the user prefers Python" saves it again, and after a month the store holds the same fact fifty times. Retrieval then returns fifty near-identical hits and crowds out everything else. A dedup check, itself a similarity search ("do we already have something this close?"), keeps memory a set of distinct facts rather than a pile of repeats.

Extract the durable facts, skip the ones already stored, embed and save the rest.

Retrieval, two ways

Getting memories back into a run has two shapes, and the choice is a design decision worth making deliberately.

As a tool the agent calls. Give the agent a search_memory tool and let it decide when to look. This fits memories it needs only sometimes: "have I solved this before?" is worth a lookup on a hard task and a waste on an easy one.

@tool
def search_memory(query: str) -> str:
    """Search past experience for something relevant to the current task."""
    return memory.query(query)

As a step that always runs first. Some memories should be present every time, before the model sees anything. A user's profile is the clear case: you want "this user is a marketer who prefers Python" in the context at the start of every conversation, not fetched only if the agent thinks to ask.

Doing that as a tool would make it optional, and doing it as a callback would scatter memory logic across a tool and a callback. The cleaner approach injects the relevant memories into the context as the run begins, in one place.

async def run(self, question, session_id=None):
    context = self._load_context(session_id)
    for memory in self.memory.query(question):     # always-on retrieval, up front
        context.add_event(memory_as_event(memory))
    context.add_event(user_message(question))
    return await self._loop(context)

Retrieval as a tool for occasional lookups; always-on for facts every run needs.

The loop, closed

Put the two halves together and long-term memory is a full cycle.

end of a run:    events -> extract -> dedup -> store in the vector memory
start of a run:  request -> retrieve relevant memories -> add to the context -> run

The agent finishes a task and quietly files away what it learned. Next time, before it starts, it pulls back what is relevant. Over many sessions it accumulates a usable, de-duplicated store of experience, and it does it with the exact retrieval machinery this series built two chapters ago.

What to take from this

• Long-term memory is RAG aimed inward. The corpus is the agent's own experience, and retrieval is the same vector search you already built.
• The memory manager extracts, dedups and stores. Dedup is the step that matters: without it the store fills with repeats and retrieval degrades.
• Retrieve as a tool for occasional lookups, or always-on for facts every run needs, and keep that logic in one place rather than split across a tool and a callback.

That closes the memory chapter, and with it the core of a working agent: it reasons, uses tools, retrieves knowledge, manages its context, holds a conversation, pauses for a human, and learns across sessions. Everything it stores and recalls, though, is also a surface worth defending, which is where the security series goes next.