Early in this series we built a yardstick: a set of questions with answers you know, to measure whether a change helped. That works, and evaluating a full agent needs more, because an agent is more than its final answer.
An agent is nondeterministic, and it takes a path. Two runs on the same input can differ, and a right answer reached by a lucky wrong path still comes from a poor agent. So you evaluate three things at once.
Three things to evaluate
The final output. Did the delivered result meet the requirement? For a summary, did it capture the key points? This is the obvious one, the yardstick from earlier, and it is necessary but not sufficient.
The components. Did the individual pieces do their jobs? Did retrieval return relevant chunks? Did the tool call produce well-formed arguments? Did the router pick the right specialist? An agent can produce a decent final answer while a component underneath it is quietly broken, and that breakage will surface on the next input.
The trajectory. Was the path sensible? Did the agent go straight to the answer, or did it wander, repeat a search, call the wrong tool, and recover by luck? Two agents can reach the same answer, one in three clean steps and one in eleven messy ones, and the clean one is the better agent. The trajectory is what the trace from the last post lets you see, and it is the part that distinguishes a reliable agent from a lucky one.
Evaluate the output, the components, and the trajectory, not just whether the final answer was right.
Building a dataset
Evaluation runs on a dataset: a set of inputs, and what a good result looks like for each. It is the yardstick, grown up.
A good dataset covers the real distribution of what your agent will face, not just the easy cases. It includes the tricky inputs, the ambiguous requests, the edge cases where agents actually fail. And it is labelled with enough to judge against: not always a single exact answer, but the criteria a good response has to meet.
# each case: an input, and what a good result requires
CASES = [
{
"input": "Summarise the Q2 support trends.",
"requires": ["names the top category", "gives a volume figure", "no invented data"],
"hard_because": "several categories are close in volume",
},
]
The requires and hard_because fields carry the design. You are collecting more than inputs; each
case records what makes it a real test and what a pass actually means.
Finding the failure modes by reading traces
You cannot write good rubrics from a desk. The real failure modes come from watching the agent fail, and the method for that has a name: open coding.
Open coding is qualitative. You run the agent across your dataset, then read the traces, one by one, and write down what you see going wrong, in plain language. "It searched three times and used none of the results." "It answered before checking the file." "It picked the summariser when the request needed code." No categories imposed up front; you let the failure modes emerge from what the traces actually show.
Open coding works best with more than one kind of person: the engineers who built the agent, the domain experts who know what a good answer looks like, and the QA people who own quality. Each sees failures the others miss. The output is a catalogue of how this agent actually fails, grounded in real traces rather than imagined problems.
Turning failures into rubrics and metrics
Once you know the real failure modes, you turn them into rubrics: explicit criteria for what a good response requires, derived from how the agent actually goes wrong.
If open coding showed the agent inventing figures, a rubric criterion becomes "every figure traces to the source data." If it showed premature answers, a criterion becomes "the agent gathered all required inputs before answering." The rubric is the failure catalogue, turned into things you can check.
| From open coding | To a rubric criterion | Measured as |
|---|---|---|
| Invents figures not in the data | Every figure traces to a source | A grounding check per figure |
| Answers before gathering inputs | All required inputs gathered first | A trajectory check on the path |
| Picks the wrong specialist | Routes to the correct agent | A component check on the router |
Read traces to find real failures, turn each failure into a rubric criterion, measure it.
Rubrics grounded in real traces beat rubrics written from imagination every time, because they measure how this agent actually fails rather than how you guessed it might.
What to take from this
- Evaluating an agent is not just checking the final answer. Because an agent is nondeterministic and takes a path, you evaluate the output, the components, and the trajectory.
- Build a dataset that covers the real distribution, including the hard cases, labelled with what a good result requires rather than always one exact answer.
- Find the real failure modes by reading traces (open coding), with more than one kind of reviewer, then turn each failure into a rubric criterion you can measure.
You now have a dataset and rubrics: inputs, and criteria for a good response. Checking a response against a rubric by hand does not scale, though. The next post uses a model to do the grading.