Model comparisons usually come down to two numbers: a benchmark score and a context window. Both fit neatly on a slide, and both leave out what decides whether the thing can run an agent.
Four capabilities settle that question. Three are obvious once stated. The fourth belongs to your architecture rather than to the model.
The four that matter
Tool calling. The model has to reliably choose a tool and produce well-formed arguments for it. A model that struggles here blocks everything downstream, because a tool call is how an agent acts at all.
Structured output. Output that other software consumes has to arrive as a schema. A schema is what lets the agent sit inside your system, feeding real code real values.
Long context support. The window has to hold the conversation, the tool results and the retrieved documents at once.
Swappability, which lives in your code. New models land monthly and rankings move, so the design should assume the model will change. Route every call through one provider-agnostic layer, and a vendor becomes a line of configuration.
| Capability | What to actually test | Failure you will see |
|---|---|---|
| Tool calling | Well-formed arguments under messy input | Invented parameters, wrong tool chosen |
| Structured output | Schema conformance under pressure | Valid-looking JSON with the wrong shape |
| Long context | Whether it uses the middle of the window | Detail in the middle quietly ignored |
| Swappability | Same code, second provider | A rewrite, not a config change |
The four things that decide whether a model can run an agent.
The claim worth stopping on
Modern models advertise windows from hundreds of thousands of tokens up to a million, and that number gets treated as the headline spec.
Quality comes from putting the right information in the window, at the right time, in the right form. Size describes the container, and the contents do the work.
The window gets sold the other way round. A bigger number reads like headroom, as though someone had handed you more memory.
What you actually receive is a larger container, and the job of deciding what goes into it stays with you.
What a bigger window costs
Recall how an agent holds a conversation: the model keeps nothing between calls, and everything it appears to remember gets re-sent every time.
Re-sending changes what a context window is. The window measures the payload you re-transmit on every single turn of the loop.
So a bigger window brings three consequences, and one of them is good.
The good one: you can fit more of the genuinely relevant material in.
The first cost: money. You pay for those tokens on every turn. An agent that runs 20 steps sends the whole accumulated context 20 times, so filling a large window because the room exists creates a recurring bill.
The second cost: noise. Retrieval that returns 40 documents when 3 were relevant buries the useful ones. The failure stays quiet, which makes it the worst kind. The agent keeps running, answers slightly worse, and gives you no signal about why.
Three questions worth asking a vendor
Leave "how big is the window" aside and ask these three instead.
Does it use the middle? Long-context performance varies across the window. Put a decisive detail in the middle of a long context and see whether the answer reflects it. Cheap to test, and more informative than the advertised number.
What does a 20-step run cost? Ask for the realistic cost of one full agent run, with context accumulating each turn, and treat the price per million tokens as an input to that figure.
How hard is it to swap? An answer that involves rewriting the agent tells you the model is a dependency. One adapter layer, with the provider chosen by configuration, turns it back into a component.
Testing them, rather than reading the spec sheet
All four are testable in an afternoon, and the tests run on plain Python. Run them before you write your question set, so that when the set exists you are already measuring a model you trust to hold a schema. Here is what I would actually run.
Tool calling, under messy input. Feed it the way people really type, rather than a clean sentence.
TOOLS = [{
"type": "function",
"function": {
"name": "get_weather",
"parameters": {
"type": "object",
"properties": {"city": {"type": "string"}},
"required": ["city"],
},
},
}]
reply = client.chat(model=M, tools=TOOLS,
messages=[{"role": "user", "content": "whats the weather like in dubai tmrw"}])
# Pass: exactly one call, name == "get_weather", arguments parse as JSON, city == "Dubai"
# Fail: no call at all, an invented parameter such as "date", or city == "dubai tmrw"
The third failure matters most. A model that stuffs the whole phrase into one field looks like it worked, right up until the downstream API rejects the call.
Long context, in the middle. Almost nobody runs this test, and it tells you what the advertised number leaves out.
needle = "The internal project codename is Kingfisher."
def probe(model: str, filler: str, position: float) -> str:
cut = int(len(filler) * position)
haystack = filler[:cut] + needle + filler[cut:]
return ask(model, haystack + "\n\nWhat is the internal project codename?")
for pos in (0.1, 0.5, 0.9): # start, middle, end
print(pos, probe(M, filler_50k_tokens, pos))
Run it at 10, 50 and 90 percent. If the middle answer degrades while the ends hold, you have learned something the spec sheet will never tell you, and you have learned it about the exact region where a long agent conversation puts its history.
Swappability. The test stays embarrassingly simple, which is the whole point.
for model in ("provider-a/big", "provider-b/big"):
run_my_agent(model)
A one-line change means you hold a component. Touching the agent to make it work means you hold a dependency, and your next model choice becomes a migration project.
Structured output, at volume. Send the same request fifty times against a schema with a typed field, and count how many replies parse and carry the right types. A model that passes forty-eight times out of fifty still breaks your pipeline twice a day once traffic is real.
Most model problems turn out to be context problems
Most of what people call an agent problem traces back to context. The model performed as designed, and it was handed the wrong things, in the wrong order, in a window someone filled because the room existed.
Picking a model with a big window and no plan for what goes in it works like buying a bigger van because you keep losing your tools. The van was doing fine.
Right information, right time, right form. The window is the constraint that forces you to decide what those are, and a bigger one lets you postpone the decision at a higher monthly cost.
What to take from this
- Test tool calling, structured output, long-context recall and swappability. The benchmark score and the window size cover none of them.
- The window is the payload you re-send every turn, so its size sets a recurring bill.
- Route every call through one provider-agnostic layer, and changing model stays a config change.