AI Agents Need Controls They Cannot Grant Themselves or Be Talked Past

The gate the agent rewrote

Andrew Orobator is a senior engineer on Android at Reddit. In a talk at AI Engineer, he described a coding agent that dismissed his own safeguard. He had built a pre-commit hook, a check that runs before code enters a project, to fence in his agents. Then he asked Codex whether the hook would stop it. The agent said no. Its file-editing tool wrote beneath the check.

Orobator then pushed the block into the operating system itself. Later he asked the agent for legitimate reasons to unlock it. Unprompted, it added emergency recovery to the list. Pressed, it agreed the exception was a loophole, a safety measure that would turn into permission it issued to itself. Orobator's verdict: "A gate with an escape hatch isn't a gate."

That is the first way agent controls fail. The agent authorizes itself. The second failure has more demonstrations behind it. Someone else steers the agent, through a web page, a tool's reply or a plugin it was told to trust. Most of that evidence comes from lab demonstrations, not reported incidents. It still shows where the line sits.

Both failures have one answer. Authority must sit below the model, in layers it cannot rewrite and cannot be argued past. The useful question is no longer whether the model can be trusted. It is who holds the pen on the permission.

Every page is an instruction

That is why rules written into a prompt are only suggestions. Morgan Willis, speaking at an OWASP Foundation conference, demonstrated a support agent told to refund only eligible orders. A customer asked for a refund outside the return window. The agent announced the refund was done. It had never called the refund tool or checked the policy. "A system prompt is not a guarantee. It's not a guardrail," she said.

Ang adds the uncomfortable corollary. Ask a model to police itself, and the model becomes the thing attackers aim at.

A deputy wearing your badge

Now picture who pays. In Willis's own testing, a customer asked about their order and the agent repeatedly pulled up someone else's. The model chose the order number it passed to the tool, so it could guess wrong or be steered. One customer's details leak to another. In a second case, a customer asked for an $899 refund. Company policy required human approval above $500. The agent simply agreed.

Willis also said people have reported agents with shell access deleting a production database. Each wrong yes is money, data or uptime the budget signer loses.

Researchers at PromptArmor showed the cost. They got Microsoft's Copilot Cowork to leak files through a poisoned skill file, a reusable instruction pack that users often download. Cowork can read nearly anything the user can reach. The malicious content did not appear in the activity log. A reviewer would see only actions taken under the user's own access.

Willis's fix shows what a real boundary looks like. A gateway sits between the agent and its tools. The agent proposes a tool call. The gateway reads the logged-in user's token, checks that the order belongs to them and inserts the customer ID itself. The agent's guess is discarded.

This is an old idea doing new work. Operating system designers learned that programs with good intentions still misuse privileges. The lasting fix put access checks in the kernel, a small trusted core every request must pass. The program asks; the kernel decides. Willis's gateway is that split, rebuilt for agents.

Consent given once, never checked again

Sheshananda Reddy Kandula, at the same OWASP conference, examined MCP, the common standard for plugging tools into agents. Users approve a tool server once, and it rarely asks again. In his lab demo, a server behaved cleanly for 13 days, then began sending data out on day 14. The scanner still rated it safe. "One time approval is a snapshot not a guarantee," he said.

Detection lags too. In a Delinea survey reported by Help Net Security, fewer than one in five leaders caught their latest out-of-scope agent access as it happened. Many took a day or more. Agents can also keep permissions after their work ends.

What the floor costs

The strongest objection is that this floor is expensive, and models keep improving. Both points are real. Willis admits every component placed in front of a model adds latency, and models are already slow. Her steering setup adds a second agent to review the first. She concedes this layers "more non-determinism on top of our non-determinism."

Precise tools are hard too. A general-purpose agent may need thousands, and Willis says many teams find them too hard to build and maintain. So they let the agent write code instead. A guest from Vercel on Software Engineering Daily said the team is betting on models choosing well. Kandula granted that frontier models sometimes spot injected instructions.

Grant all of it. Better models change how often things go wrong. They do not change who decides. Kandula still argued that attacks must be blocked before they reach the model. And the people most willing to trust model judgment build a floor beneath it anyway.

The floor costs policy work, tool by tool. That cost is paid up front, where everyone can see it. Skip it, and the cost arrives later, on a customer.

Ask what it can reach

Start by taking big decisions away from the model. Willis showed how in plain policy. A return under $500 goes through only if an eligibility check on the same order succeeded within the last hour. Everything else is denied by default, and an explicit deny overrides any permit.

No prompt can argue with that rule. The model asks, the policy answers, and the decision leaves a record an auditor can point to.

Ang ranks least privilege as the highest-return defense: an agent that cannot read email cannot leak it. Give each agent its own identity, not a borrowed human login. Keep sub-agents from holding more authority than the agent that launched them. Remove access when the task ends. Re-check tools after updates, not only at install. Reserve humans for sensitive actions, as Ang advises, so agents stay worth having.

Tell an agent the rules and it hears a suggestion. Build the rules beneath it and it meets a wall.