Why an agent needs a knowledge base

The agent we built can search the web. Plenty of real questions have answers the web never held: an internal wiki, a project's code, a spreadsheet a customer attached to a ticket.

Getting that private data into the context is the whole of this chapter, and the approach scales in three clean steps.

The rule underneath all of it

Retrieval follows the same rule as web search, which we met early in this series. In a large pile of data, find only the parts relevant to the question and put those in the context.

The reason is the memory post from earlier: the context window holds a payload you re-send on every call, and it has a limit. So the goal is never "give the agent all the data". The goal is "give the agent the right slice", and how you find that slice depends on how much data there is.

Step one: a single file fits

Start with the easy case. A ticket arrives with a CSV attached and the question "which month had the highest volume?"

The file is a few hundred rows, so it fits in the window. Read the whole thing, put it in the context, done.

@tool
def read_file(path: str) -> str:
    """Read a text or CSV file and return its contents."""
    return Path(path).read_text()

One tool added to the agent from chapter 4, and a whole class of "answer from this attached file" questions is solved. No retrieval machinery needed, because the data already fits.

Step two: many files need exploration

Now the attachment is a zip. Extract it and you get a tree.

project/
├── README.md
├── requirements.txt
├── src/
│   ├── main.py
│   └── database.py
└── config/
    └── settings.json

The question is "what database does this project use?" The answer is in requirements.txt, or database.py, or settings.json, and you do not know which in advance.

Reading every file into the context works for five files and falls apart at five hundred. The better move is the one a developer makes in a new codebase: look at the structure first, use the file and folder names as clues, open the promising ones, and follow the trail.

That is structure-based search, and it is exactly how tools like Claude Code and Cursor read a project. A later post builds the filesystem tools that let an agent explore this way.

Step three: too much data needs retrieval

Structure-based exploration has a ceiling too. A single log file with fifty thousand lines does not fit the window, and its structure gives no clue where the answer sits. A company wiki with thousands of loosely organised documents is too much to walk one by one.

At this scale you need a mechanism that pulls out only the relevant parts. For "how do we connect to the database?", you do not want fifty thousand lines of code in the context. You want the handful of lines about the database connection, and nothing else.

Finding that handful, by meaning rather than by reading everything, is what vector search and keyword search are for. Retrieval-augmented generation, RAG, is the name for the whole approach: retrieve the relevant slice, then let the model generate an answer from it.

As the data grows, the approach scales: read it, explore it, retrieve from it.

The three cases, side by side

Read the middle column top to bottom. The work grows only when the data forces it. A small attachment never needs a vector database, and reaching for one is a common way to overcomplicate a simple job.

What to take from this

• Retrieval follows one rule: put the relevant slice in the context, not all the data, because the window is a payload with a limit.
• The approach scales in three steps. A small file you just read. Many files you explore by structure. Too much data you retrieve from by meaning.
• Match the approach to the data. The heavy machinery earns its place only at the top of that ladder.

The rest of this chapter climbs the ladder. The next post lays out the search methods that make the third step possible, and which one to reach for when.