Two things separate a working demo from a working system

The demo always works. One call, a good answer, everyone is impressed.

Then you try to put it inside software, and two problems appear that have nothing to do with how clever the model is. This post is those two problems, with the code.

Problem one: prose is not an interface

Ask a model to classify a support ticket and you get something like this.

Sure! This looks like a billing issue. I would call it fairly urgent
given the customer mentions a failed payment. Let me know if you need
anything else.

Correct, and useless. Your next line of code wants ticket.urgency, and what arrived is a friendly paragraph with the urgency buried inside an English sentence.

You can regex your way out of that, and people do, and it breaks the first time the model phrases it differently. The real fix is to ask for a shape instead of prose.

from pydantic import BaseModel

class Ticket(BaseModel):
    category: str      # billing | technical | account
    urgency: int       # 1 to 5
    summary: str

The class above is a contract. You send it with the request, and the reply comes back shaped to fit.

reply = client.chat(
    model=MODEL,
    response_format=Ticket,          # the reply must fit this shape
    messages=[{"role": "user", "content": ticket_text}],
)

ticket = Ticket.model_validate_json(reply.content)
print(ticket.urgency + 1)            # an int. arithmetic works. no parsing.

Now the model's output is an object, and the agent can sit inside your system rather than beside it.

The half people skip

Here is the part that separates people who have shipped this from people who have demoed it.

The model will break the contract. Not often. Often enough.

A reply will come back with urgency: "high" where you asked for an int. Another will invent a category you never listed. A third will arrive as valid JSON in the wrong shape, which is the worst case, because valid JSON looks like success.

Validation is load-bearing for that reason, and what your code does on a failed parse is your system.

from pydantic import ValidationError

try:
    ticket = Ticket.model_validate_json(reply.content)
except ValidationError:
    # This is not an edge case. Decide, now, which of these you are:
    #   retry once with the error fed back in
    #   fall back to a safe default and flag for a human
    #   fail closed and raise
    ticket = escalate_to_human(ticket_text)

An agent without that except branch does not have fewer failures. It has the same number of failures, arriving somewhere less convenient.

And note what the schema bought you beyond the parsing: an unlisted category is now a caught error rather than a silently wrong record. category: str is weaker than it should be for that reason. An enum would be better, and the tighter the type, the more the validator catches for free.

A schema is a contract, and the interesting question is what happens when it is broken.

Problem two: it is slower than you think

Your demo made one call and it took about a second. No problem.

Then the work arrives in batches. Classifying last quarter's 412 tickets means 412 calls. Running your twenty test questions against a model means twenty. Neither of those is an unusual ask, and both turn one comfortable second into a number worth measuring.

Take the twenty.

results = []
for ticket in tickets:                    # 20 tickets
    results.append(classify(ticket))      # ~1.5s each
# about 30 seconds

Thirty seconds, and the machine sat idle for most of them. Every one of those calls was your process waiting on a network round trip. Waiting is the bottleneck here, and waiting is exactly what async was built to overlap.

import asyncio

async def classify(ticket: str) -> Ticket:
    reply = await client.chat(model=MODEL, response_format=Ticket,
                              messages=[{"role": "user", "content": ticket}])
    return Ticket.model_validate_json(reply.content)

results = await asyncio.gather(*(classify(t) for t in tickets))
# about 2 seconds

Same 20 calls, same total work for the provider, roughly one call's wall clock for you, because they overlap.

(Illustrative, from ~1.5s per call. Your numbers will differ.)

The third row is the one that bites

Run that gather over 500 tickets and it stops working. Every provider caps requests per minute and tokens per minute, and firing 500 concurrent calls is the fastest way to find out what yours are.

The fix is one object.

LIMIT = asyncio.Semaphore(5)              # at most 5 in flight at once

async def classify_limited(ticket: str) -> Ticket:
    async with LIMIT:                     # wait here if 5 are already running
        return await classify(ticket)

results = await asyncio.gather(*(classify_limited(t) for t in tickets))

A semaphore is a counter with a queue. Five calls go through, the sixth waits until one finishes. You keep almost all of the speed and stop hitting the ceiling.

Pick the number from the provider's published limits, not by trial and error, and remember that both limits apply. You can be well under the requests-per-minute cap and still be throttled on tokens, because a long conversation carries a much bigger payload per call. Which, if you read the previous post in this series, is a problem that gets worse every turn.

Both problems are the same problem

A demo is one call working once. A system is many calls working repeatedly, and the difference is entirely in what you built around the call.

The schema is how the output joins your software. The semaphore is how the calls survive contact with a real provider. Neither is about the model, and neither gets easier if you pick a better one.

Keep that schema in mind, because it comes back later as the mechanism behind tool calling: the same idea pointed the other way, letting the model ask your code to go and do something.