Building an agent is a loop: ship a version, watch it fail, work out why, fix that, repeat. The loop only turns if you can tell whether the last change made things better.
Without a fixed measuring stick, you get opinions. The agent feels sharper today. The new prompt seems to handle edge cases better. Those sentences are how teams spend three weeks moving sideways.
So the first artefact worth building is the yardstick, and it comes before the agent.
What a good yardstick looks like
A yardstick here is a set of questions with answers you already know. The properties that make it useful are specific.
The answers are unambiguous. "Which region had the highest ticket volume?" has one answer you can
compare against with ==. "How is our support doing?" has no gradeable answer, so it teaches you
nothing about whether yesterday's change helped.
Most questions need more than one step. A question answered by a single lookup measures the lookup. A question that needs a document read, a figure found, and arithmetic over both measures the thing you are actually building.
Anyone can check the work. You want to look at a failure and see immediately where it went wrong. Questions requiring specialist judgement to grade turn every failure into a debate.
They resemble the work. Twenty puzzles about trivia will make your agent good at trivia.
The public version, and why it exists
A benchmark called GAIA collects exactly this kind of task. Meta and Hugging Face released it in 2023, and it consists of question-and-answer pairs where each question needs several steps: a web search, a file read, a calculation, or some combination. The answers are short and checkable.
GAIA is useful to learn on for three reasons. Clear answers give you feedback in seconds rather than after an argument. The difficulty range is wide enough that a weak agent scores badly and a good one scores well, so the number moves as you improve. And the questions need no specialist background, so you can read a failure and understand it without being a doctor or a lawyer.
Newer and harder benchmarks exist. For learning how agent development actually feels, the difficulty of GAIA's easier problems is close to ideal.
Why your own twenty are worth more
A public benchmark has one structural weakness, and it is worth understanding before you lean on any published score.
Public benchmarks are built from public material: web pages, papers, reference sites. Models are trained on large slices of that same public material. So a model can arrive already holding some answers, and a score partly measures what it memorised rather than what it can work out.
Questions about your own systems have no such problem. Nobody trained on your ticket export. A model that answers a question about your quarter did so by reading what you gave it, which is exactly the behaviour you want to measure.
| Public benchmark | Your own twenty | |
|---|---|---|
| Setup cost | Download it | An afternoon |
| In the training data | Possibly | No |
| Resembles your work | Rarely | Exactly |
| Comparable to others | Yes | No |
| Good for | Positioning a model | Deciding if YOUR agent improved |
Use a public set to get a rough sense of a model. Use your own to decide whether your agent got better, because that decision is the one you make every day.
A public set measures the model. A private set measures your agent.
Writing the twenty
Take the running question from earlier in this series:
Our support team closed 412 tickets last quarter. If we route the routine ones to an agent, how many staff-hours would that free, and does the saving cover what the model calls cost?
That is one question. You need nineteen more of similar shape, and the format is deliberately plain.
# Each case: the question, the answer you already know, and what it takes to get there.
CASES = [
{
"question": "How many of last quarter's 412 tickets were password resets?",
"answer": "118",
"needs": ["file"],
},
{
"question": "What does one million input tokens cost on our current model today?",
"answer": "2.50",
"needs": ["web"],
},
{
"question": "At 6 minutes per password reset, how many staff-hours did they consume?",
"answer": "11.8",
"needs": ["file", "calculation"],
},
]
The needs field is the part people leave out, and it does more work than the rest of the file.
Labelling each question with the capability it requires lets you answer a question about the answers: how many of these are reachable at all with the tools this agent currently has? A set where sixteen of twenty need a web search tells you something useful before you run anything, and it tells you the same thing whichever model you plug in.
The cycle it unlocks
With the set fixed, development becomes mechanical in the good sense.
- Run all twenty. Record what passed.
- Read the failures, one by one, and name the cause. Wrong tool chosen. Right tool, unusable output. Ran out of steps. Gave up.
- Change one thing.
- Run all twenty again.
Step two is where the actual learning happens, and it needs failures you can read. That is why the answers have to be unambiguous and the questions have to be checkable by a normal person: you will spend far more time reading failures than admiring the score.
Run, read the failures, name the cause, change one thing, run again.
Change one thing at a time. Two changes and a moved number tell you nothing about which one moved it.
What to take from this
- Build the measuring stick before the agent. Without one, "it feels better" is the best evidence you will ever have.
- Good questions have unambiguous answers, need several steps, and can be checked by anyone.
- Twenty questions about your own data beat any public leaderboard for the decision you make daily: did my change help?
Twenty questions and a scoring loop give you a number. The interesting part is what that number does when you take the agent's tools away.