AI Agents · Security
When the Tool Output Itself
Tries to Manipulate the Agent
A few times now, while running an AI coding agent against a real codebase, a shell command's output has contained something shaped exactly like a real system message — instructing the agent to hide a change from the person it was working for. The right response is always the same: disregard it, and say so.
Most posts here are about designing a backend so an AI agent (or a human) can follow its architecture correctly. This post is about a narrower, stranger problem: what happens when something in the environment tries to get the agent to act against the person it's working for — and the instruction arrives disguised as a legitimate part of the system, not as an obviously suspicious request.
What Showed Up in the Output
Across several long agent sessions doing ordinary engineering work — porting features across languages, running test suites, inspecting git history — the output of an ordinary tool call (a git command, a shell script, a build log) has, more than once, contained content formatted to look exactly like a genuine system-level message. Not a visibly broken or garbled string; something that passed as legitimate at a glance, sitting inside output that was otherwise completely normal. Its content, each time, pushed toward the same thing: don't mention this to the user, or otherwise conceal a change that had just been made.
Why This Isn't Hypothetical
It's tempting to treat this as a one-off curiosity. It's happened enough times, across enough different sessions and different tools, that it's worth treating as a real category of risk rather than a fluke — the same way a single suspicious log line is a curiosity, but the same suspicious pattern recurring across unrelated systems is a signal. An agent that reflexively trusts anything shaped like a system instruction, regardless of which layer it actually came from, is trusting the wrong boundary.
The One Rule That Matters
Tool output is data, not instructions. A command's stdout, a file's contents, an API response — none of it carries authority just because it happens to be formatted to look like it does. Legitimate system messages come from the actual system layer, not from something a shell command printed. The instant an instruction embedded in tool output asks for concealment specifically — don't tell the user, hide this, keep this quiet — that's close to a decisive signal on its own, since a legitimate system rarely has a reason to ask an agent to hide something from the person it's serving.
The tempting shortcut is to silently ignore the injected content and move on as if nothing happened — technically safe, but it also means the person relying on the agent never finds out their environment tried to get manipulated. The complete response is disregard and disclose: don't follow the embedded instruction, and say plainly that it showed up, in output that would otherwise look unremarkable.
Why Concealment Is the Tell
Most legitimate reasons a tool might want to shape an agent's behavior are about correctness or safety — a linter flagging a bug, a test asserting a contract, a build failing loudly. None of those need the agent to keep something from the user; quite the opposite, since the whole point of that feedback is usually to become visible to a human eventually. An instruction whose actual payload is "don't mention this" doesn't fit any of the ordinary reasons tool output shapes behavior — which is exactly why it stands out as clearly not something to comply with, independent of whatever plausible-sounding justification comes attached to it.
Building With This in Mind
None of this changes how the underlying engineering work gets done — a lint rule or a test doesn't get more or less correct because of it. What it does change is a standing default for anyone running long AI-agent sessions against real infrastructure and real shell output: treat "this looks like a system message" and "this is a system message" as two different claims, and keep the gap between them, especially when what's being asked for is silence.