Your system prompt is not a security boundary

"It cannot do that. We told it not to in the system prompt."

I keep hearing versions of that sentence. It describes a policy. It does not describe a control, and the gap between those two words is most of what agent security is about.

The advice itself is right

A system prompt does four jobs. It defines the agent's identity, specifies output format and style, sets boundaries on what it must not do, and states the limits of its knowledge.

The third one is where most guidance is thinnest, and it matters most. An agent acting autonomously, with no stated boundaries, will do things you never intended. Defining what not to do is at least as important as defining what to do.

So a support agent gets a prompt like this. It may look up an order and issue a refund. It may not close a customer's account.

That is how nearly everyone builds it, and it is correct as far as it goes. It is also where the confusion starts.

What the model actually receives

Every call, the model is handed something like this.

messages = [
    {"role": "system", "content": "You may issue refunds. You may not close accounts."},
    {"role": "user",   "content": user_text},           # whoever is talking to the agent wrote this
    {"role": "tool",   "content": retrieved_document},  # whoever wrote the document wrote this
]

Three different authors. One buffer.

Now look for the field that marks which parts are trusted.

There is not one. role is a label that helps the model interpret structure. It is not a privilege level, it is not enforced by anything, and it does not travel with the text as a permission.

Different authors, one buffer, and nothing in it marks which parts are trusted.

The model does not experience your system prompt as law and the retrieved document as data. It sees a sequence of tokens and weighs them. Your instruction sits first in the buffer. First is not the same as privileged, because privilege is not a property that buffer has.

A request that usually works

So "we told it not to" is a request. It usually works, and that is precisely what makes it dangerous.

A control that fails loudly gets fixed. A request that holds most of the time gets trusted, then built on, then cited in a risk register. It holds right up until some later text in the buffer is more persuasive than your instruction, and nothing announces the moment that happens.

I am not going to publish a working way to do that, and you do not need one to see the problem. The structural point is enough: if the only thing standing between your agent and an irreversible action is a sentence competing with other sentences, you are relying on the model's judgement, every call, forever.

Policy or control

Auditors have been asking a version of one question for decades. Show me the control. Not the policy document, the control. Who cannot do this, and what stops them?

A boundary in the prompt is a request. A boundary in the tool layer is a fact.

The left column is a policy. The right column is a control. Both are useful. Only one of them survives the question.

Where the boundary actually goes

The fix is unglamorous and it is not an AI problem.

# The agent cannot close an account because it was never handed the ability to.
TOOLS = [search_orders, issue_refund]

# And the expensive path it does hold is gated outside the model.
def issue_refund(order_id: str, amount: Decimal):
    if amount > REFUND_LIMIT:
        return await_human_approval(order_id, amount)
    return refunds.create(order_id, amount)

Two moves, both boring, both auditable.

Do not hand it the capability. An agent cannot call a function that is not in its registry. This is the strongest boundary available and it costs nothing, because most agents are given far more reach than their task needs.

Gate the expensive path in code. For the tools it must hold, the threshold check lives in the function, not in a sentence. Then the approval is a record, not an intention.

Notice what changed. The boundary stopped depending on the model agreeing with you.

What this touches

For the frameworks, this sits on the MAESTRO Agent Frameworks layer, and reads as T6 Intent Breaking and Goal Manipulation (redirecting the agent's stated objectives) escalating into T3 Privilege Compromise (acting beyond the permissions intended for it). In the OWASP Agentic Top 10 it is ASI03 Identity and Privilege Abuse.

On the regulatory side, and I mean this as scope rather than a citation: this is the segregation of duties and authorisation area that every GCC rulebook already covers for human staff. Those obligations were written assuming a person needs a permission to do a thing. An agent with a tool has the permission. Which obligations actually bite depends on your regulator and your architecture, and that mapping is an advisory estimate until someone checks it properly.

The part worth keeping

None of this makes the prompt useless. It is the right instrument for behaviour and quality, and a well written boundary produces a better-behaved agent.

It is just not a security boundary. Treating it as one is how you end up holding a policy where you believed you had a control.

Write the boundary in the prompt, by all means. Then go and build it somewhere the model cannot argue with it.