Way back in this series we built a yardstick: twenty questions with answers you already know, each labelled with the capability it needs. Then we ran a model against it with the tools switched off, and watched it hit the ceiling, because most of the questions could not be answered from training data alone.
We now have an agent. Time to switch the tools on and measure the difference, on the exact same twenty.
The setup is one line different
The whole point of building the agent properly is that this comparison costs almost nothing to run. The questions are the same. The scoring is the same. The only change is whether the agent has tools.
tool_free = Agent(model=client, tools=[], instructions=SYSTEM)
with_tools = Agent(model=client, tools=[read_tickets, search_web, calculate], instructions=SYSTEM)
for label, agent in [("no tools", tool_free), ("with tools", with_tools)]:
correct = await score(agent, CASES) # the yardstick from earlier
print(label, correct, "/", len(CASES))
Two agents, same questions, one difference. That is the experiment.
What the two runs show
Run it on questions that genuinely need a lookup or a file, and the two columns separate sharply.
| No tools | With tools | |
|---|---|---|
| Questions solved (of the tool-needing ones) | a handful | most of them |
| Where the answers came from | training data, when it happened to know | the actual file and the live page |
| Cost per run | low | several times higher |
| A wrong answer's source | a confident guess | traceable to a fetch |
The solve rate jumps, and that is the headline everyone expects. The tool-free run was capped by construction: no file access, no search, so the questions needing those were unreachable however capable the model. Hand the same model the tools, and the questions it could not reach become questions it can.
Same model, same questions. Tools turn unreachable questions into answered ones.
The second number the floor run could not show
The tool-free run had nothing to say about cost, because it made one call per question. The agent makes several: reason, call a tool, read the result, reason again. Each pass is another model call, and the context grows each time, so it carries more tokens than the last.
So the with-tools run costs several times more than the tool-free run, and the multiple is large enough to plan around. The jump in capability comes with a jump in spend, and the number is worth seeing before you ship rather than after the first invoice.
The same arithmetic from the memory post now has real consequences: an agent that loops ten times over a growing context pays for that context ten times. Tools give you capability you pay for per step, rather than capability that comes free.
| Single model call | Agent with tools | |
|---|---|---|
| Calls per question | one | several |
| Context per call | fixed | grows each step |
| Capability | capped at the ceiling | reaches the tool-needing questions |
| Cost | low, flat | higher, and scales with steps |
Tools lift the ceiling and raise the bill. Both numbers are worth measuring before you ship.
Why the comparison is the deliverable
A single number, "the agent got most of them right", tells you little. The difference between the two runs is the thing worth knowing, because it answers the question you actually have: is the extra cost and latency of an agent buying you enough capability to be worth it?
For questions that genuinely need a lookup, the answer here is clearly yes: a handful solved becomes most solved. For a task where the tool-free model already did fine, the same comparison would show a small gap and a large bill, and it would be telling you to build a workflow instead. That is the decision from the very start of this series, now backed by a measurement rather than a guess.
What to take from this
- Because the agent was built properly, switching tools on and off is a one-line change over the same yardstick. The comparison is nearly free to run.
- Tools lift the solve rate on tool-needing questions from a handful to most, and they give every answer a source instead of a guess.
- Tools also multiply the cost, because an agent makes several calls over a growing context. Measure both numbers, and let the difference decide whether an agent is the right build.
The arc this series opened now closes. A model on its own hits a ceiling. Tools lift it, at a price. An agent is the loop that wields them, and you now know how to build one, from the state it keeps to the number that tells you it was worth it.