An evaluation you run occasionally and glance at changes nothing. Evaluation earns its keep when it sits between a change and production, and decides whether the change ships.
The last piece of building an agent is a simple principle with a real consequence: without quality verification, there is no deployment.
The eval gate
You already gate code on tests. A change that breaks a unit test does not merge. An eval gate applies the same rule to agent quality: a change that drops the evaluation score does not ship.
Concretely, the evaluation harness runs inside CI. When someone opens a change to the agent, its prompt, a tool, the model, the harness runs the dataset through the new version and scores it. If the score falls below the bar, the gate blocks the merge, the same way a failing test does.
open a change to the agent
-> CI runs the eval harness on the dataset
score >= bar -> merge allowed
score < bar -> blocked, with the failing cases shown
The effect is that a regression is caught before it reaches a user, not after. Someone tweaks a prompt to fix one case and quietly breaks five others; the gate sees the five and stops it. Without the gate, that trade ships and you find out from complaints.
A change runs through the eval harness in CI; a drop in score blocks the merge, like a failing test.
Stages of the gate
The gate is a few checks, tightening as a change moves toward production.
A fast subset runs on every change, the first line of defence, cheap enough to run on each commit. A fuller evaluation runs before a merge to the main line. And a broad run, against the largest dataset, gates the actual deployment. Each stage trades cost against coverage: quick and frequent early, thorough and slow before release.
The point is that quality is verified continuously, at every step where a change could slip through, so "we tested it once" never becomes "we shipped a regression."
The flywheel: evaluation is also a product
Here is the part that keeps the whole thing alive. An evaluation set written once and never touched goes stale, because the agent's world changes: new kinds of requests, new failure modes, new edge cases. A gate guarding a stale test set gives false confidence, passing changes that would fail on today's real inputs.
So evaluation is a product you keep improving, never a one-time artefact, and the loop that improves it is the quality flywheel.
- The agent runs in production, and you observe it with the telemetry from earlier in this chapter.
- You review the real operational data for failures the test set never anticipated.
- You fold those into the dataset and rubrics, so tomorrow's gate catches today's real failures.
- The better gate protects the next change, and the agent improves, which surfaces new failures, and round it goes.
Observe production, find new failures, fold them into the dataset and rubrics, and the gate keeps getting sharper.
The flywheel is why a good evaluation set is never finished. Every real failure you feed back makes the gate a little sharper, which makes the next release a little safer, which is the difference between an evaluation that decays and one that compounds.
The loop this whole series has been
Step back, because this closes something. The series opened by building a yardstick: a set of questions with answers you know, run to see if a change helped. Everything since, tools, memory, planning, code execution, multi-agent, has been building the agent. This last chapter puts the yardstick back at the centre, grown into a dataset, a judge, and a gate.
The shape is the same one from the very first eval post: run it, read the failures, name the cause, change one thing, run again. What changed is the scale. That afternoon yardstick has become a continuous system that observes a live agent, learns from how it fails, and refuses to ship a version that got worse. Build the agent, measure it honestly, and let the measurement decide what reaches a user. That is the whole discipline, start to finish.
What to take from this
- Evaluation pays off when it gates deployment. An eval harness in CI blocks a change that drops the score, the way a unit-test gate blocks a broken build, so a regression is caught before a user sees it.
- Gate in stages: a fast subset on every change, a fuller run before merge, a broad run before deploy, trading cost against coverage as a change nears production.
- Evaluation is a product, not a one-time artefact. The quality flywheel, observe production, fold real failures back into the dataset and rubrics, keeps the gate sharp as the agent's world changes.
The build is complete: an agent that reasons, uses tools, retrieves, remembers, plans, runs code, and collaborates, measured by a system that decides what ships. Building it is one story. Knowing every way it can break is the other, and that is the series that has run alongside this one.