All posts
Agentic AI

Everything a tool returns is untrusted input

An agent's tools exist for one reason: to reach information the model never had. A web page, a document, a database row, a colleague's ticket comment.

All of it comes back as text, and the text goes into the same list as your system prompt.

Watch the buffer fill up

Here is the loop from a tool-using agent, with the messages spelled out.

messages = [
    {"role": "system", "content": "You are a support agent. Refunds under 500 only."},
    {"role": "user",   "content": "Why was ticket T-8801 escalated?"},
]

reply = model(messages, tools=TOOLS)          # asks for read_ticket("T-8801")
messages.append(reply)
messages.append({
    "role": "tool",
    "content": run_tool(reply.tool_call),     # <- whatever the ticket says
})

reply = model(messages, tools=TOOLS)          # now decides what to do next

Look at the second call. The model is making a decision, and the material it is weighing includes a string that came out of a ticket. Someone wrote that ticket. It was not you.

Nothing in the payload marks that string as data rather than direction. role: tool says where the text came from, and it carries no authority level, because authority is not a property the message list has.

Your instructions and the fetched text arrive in the same list, with nothing marking which is which.Your instructions and the fetched text arrive in the same list, with nothing marking which is which.

The rule is forty years old

Application security has one lesson that survives every technology change: input from outside your trust boundary is untrusted, no matter what shape it arrives in.

We learned it for form fields, for uploaded files, for HTTP headers, for XML, for deserialised objects. Each time, someone had to point out that the new format was still input.

A tool result is input. It arrives from outside, its content is decided by whoever controls the source, and it flows into the component making decisions. The difference is only that the component is a model rather than a SQL parser, and the model was built to be responsive to text.

The difference is worth dwelling on. Every other parser we learned to defend was trying to be literal. A model is trying to be helpful, which means it is trying to act on what it reads.

Where the text comes from

The uncomfortable part is how ordinary the sources are.

A web page. The whole reason you gave the agent search. You control the query, and someone else controls what comes back.

A document. A PDF a customer emailed. A spreadsheet from a supplier. The agent reads it because reading it is the job.

A record someone else wrote. The free-text field of a ticket, a CRM note, a code comment. Written by a colleague, a customer, or anyone who could open a ticket.

Another agent. In multi-agent setups, one agent's output is another's input, and it is the same problem with an extra hop.

None of those require anyone to breach anything. They are the intended data flows. That is what makes this different from the vulnerability classes where the fix is "stop the attacker getting in".

What actually helps

There is no parser you can write that separates instruction from data inside natural language, and anyone selling you one is selling you a filter with a bypass. What works is the same thing that worked before models existed: stop depending on the interpretation being correct.

ControlWhat it does
Capability, not filteringAn agent that holds no destructive tool cannot be talked into using one
Inject sensitive arguments in codeThe order id comes from session state, so no fetched text can change which order
Gate the irreversible stepA threshold check in the function, needing a human, regardless of how convincing the context was
Separate read from actThe agent that reads untrusted material holds no write tools; a second, narrower path performs actions
Log the tool resultsWhen something goes wrong, the text that was in the buffer is the evidence

The fourth row is the structural one and it is worth the architectural cost. When the component that reads the internet has no ability to move money, what a web page says stops being a security question and goes back to being a quality question.

The agent that reads the world holds no write tools. A narrower path performs actions.The agent that reads the world holds no write tools. A narrower path performs actions.

What this touches

For the frameworks, this spans the MAESTRO Agent Frameworks and Data Operations layers and reads as T6 Intent Breaking and Goal Manipulation (the agent's objective redirected by material it ingested), with T2 Tool Misuse where the redirection produces a tool call and T1 Memory Poisoning where the ingested text persists into later sessions. In the OWASP Agentic Top 10 that is ASI01 Agent Goal Manipulation and ASI02 Tool Misuse, plus ASI06 for the memory case.

On the regulatory side, framed as scope rather than a citation: this lands in input validation and data integrity, which every GCC rulebook already addresses for conventional systems. The obligations were written expecting validation to be a deterministic check on a defined format. Here the input is free text and the consumer is a model, so demonstrating the control means showing the capability boundary rather than the filter. Which obligations apply depends on your regulator and architecture, and that mapping is an advisory estimate until someone checks it properly.

The part worth keeping

Tools are the right answer. The previous posts in this series argued for them, and nothing here walks that back. An agent without tools is a system that recalls instead of fetching, and fetching is what gives an answer a source you can point at.

What changes is what a tool result is. It is not a fact the agent obtained. It is a message the agent received, from a source you should name out loud when you draw the diagram.

Then design as though the message might not be friendly, because that is what every other layer of your stack already does.