Series · 27 parts
How agents break
Where an agent's new powers turn into new attack surface, and how to find the breaks before someone else does.
- 01An AI agent isn't a smarter chatbot. It's a system you handed authority toA chatbot answers your question. An agent does something about it. That one shift, from responding to acting, quietly breaks most of the assumptions your security was built on, and most teams…Read
- 02To trust an AI agent, try to break it firstThe most dangerous AI agent isn't the one someone hacks. It's the one that does exactly what you asked, in a way you never meant.Read
- 03Choosing a model is a data-residency decisionModel selection usually gets discussed in terms of the things you would expect. Capability. Cost. Latency. Occasionally someone mentions, almost in passing, that if regulation requires certain…Read
- 04Your agent asks permission for the wrong thingsThe standard advice for writing an agent prompt contains a line that is both correct and, read from a different chair, alarming.Read
- 05Your system prompt is not a security boundary"It cannot do that. We told it not to in the system prompt."Read
- 06A tool description is an instruction the model will followEvery tool an agent holds arrives as two things: a function, and a sentence describing it.Read
- 07Everything a tool returns is untrusted inputAn agent's tools exist for one reason: to reach information the model never had. A web page, a document, a database row, a colleague's ticket comment.Read
- 08What you accept when you install an MCP serverAdding a tool to an agent used to mean writing a function. Now it often means adding four lines to a config file.Read
- 09The audit trail you got for free, and what is missing from itWhen we built the agent's state, every step became one Event: an id, a timestamp, an author, and the content of what happened. The list of events is a complete, attributed, time-ordered record of…Read
- 10max_steps is a safety control, not a tuning knobOur agent's loop runs until it has an answer or it hits a step limit.Read
- 11Every tool sees everythingBuilding the tool layer, we made one decision that kept everything simple: the agent passes the full execution context to every tool.Read
- 12Human-in-the-loop is a control, until it becomes theaterEarlier in this series the argument was: do not let the model be the only thing standing between an agent and an irreversible action. The knowledge-base chapter gives that argument a mechanism.…Read
- 13Your knowledge base is an input an attacker can write toWe built retrieval so an agent could answer from private data. A query goes in, the closest chunks come back, and those chunks go straight into the context as background the model reasons over.Read
- 14An agent with a read_file tool can read any fileThe filesystem tools from the knowledge-base chapter are what let an agent explore a project: list the tree, read a file, unpack an archive. They are genuinely useful, and they share one property…Read
- 15The memory an attacker writes for youWe gave the agent long-term memory so it could learn. At the end of a run, it reads the conversation, pulls out durable facts, and stores them. Next session, it retrieves those facts and acts on them.Read
- 16What an agent forgets when it compresses its own memoryTo keep a long run inside the context window, the agent compresses its own memory. Old messages get replaced by a model-written summary, so the run can continue.Read
- 17Whose session is this? Pause, resume, and the state in betweenSessions let an agent hold a conversation across calls. Durable pause and resume let it stop for approval and continue later, from state saved in the session.Read
- 18A plan is an instruction the agent writes for itselfWe gave the agent a planning tool: it decomposes a task, writes a plan, and follows that plan for the rest of the run. The plan lives in the context, so every later step reads it and works toward it.Read
- 19Reflection is the model checking the modelReflection is a real improvement to an agent. It pauses, asks "am I on track?", and reroutes when a tool fails or the plan no longer fits. The previous chapter built it, and it works.Read
- 20Code execution is arbitrary code executionCode execution is the most powerful capability you can give an agent. In security terms it has a blunter name: arbitrary code execution, the thing every other part of your stack is built to prevent.Read
- 21A sandbox is only as strong as its boundaryThe last post argued that code execution must be sandboxed. Suppose you did it: the code runs isolated from your host, capped on resources, disposable. Step one, done.Read
- 22A skill is code you agreed to runAgent skills are how a capable agent stays organised: each skill is a folder with a document describing it and the code that does the work, loaded on demand. It is a clean pattern, and it is being…Read
- 23Exposing an agent to the network is exposing a computerA2A lets you expose an agent over the network so other systems can call it. It is genuinely useful, and it is worth stating exactly what you are putting on the network before you do it.Read
- 24When your agent calls an agent you don't controlThe last post was about exposing your agent to others. This is the mirror image: your agent calling a specialist someone else runs, a remote agent over A2A, or a third-party agent wrapped as a tool.Read
- 25Delegation spreads authority, the multi-agent confused deputyThe last two posts were about agents you do not control. This one is about your own agents, and a risk that comes from how they hand work to each other.Read
- 26Your evaluator is a model, and models can be gamedAn LLM judge is how you grade an agent at scale: a model reads a response, checks it against a rubric, and returns a score. It is genuinely useful, and this series has argued for it.Read
- 27Your agent's logs are a copy of everything sensitive it touchedTo evaluate an agent, you have to see what it did, so you capture its trajectory: every model call, every tool call and its arguments, every result, the whole context. The last few posts argued…Read