If agents fix themselves in production, someone must answer for unreviewed code

Sandhya Subramani, a senior developer advocate for generative AI at AWS, argued on the AI Engineer show that agents could repair themselves while they run, without waiting for an engineer. Her evidence was a conference demo, not a production case. Extending her point: if teams let agents fix themselves in production, code nobody reviewed would run live. Her talk leaves open who answers when that code does harm.

What was said

Subramani demoed "meta-tooling" with Strands Agents, an open-source toolkit. The agent starts with no tools. It gets a system prompt describing a good tool, plus three abilities: edit files, run shell commands and load a new tool. From there it writes its own tools, and even its own sub-agents. Her demo was a travel planner.

Her flight-booking example was a scenario, not a live system. An app is built for domestic trips. One user asks for India to Hong Kong, and the agent gives up. She asked the room: "do you just want it to fail? Do you want it to run into an error?" The other route, she said, is a hand-back to the engineering team, with a fix arriving about two weeks later.

She also warned about the other edge. "If this meta tooling or this meta agent can spin off tools and agents, it can also modify and delete." Her remedies were evals (automated checks on whether the agent used the right tool and parameters), a sandboxed execution environment, tight permissions and telemetry. She presented these as what you need before trusting such an agent in production.

Why it matters

For anyone building or buying agent software, that raises practical questions. Which actions may an agent take without approval? Who reads its changes afterwards? Who is paged when it deletes something? Subramani's guardrails cover the technical half. Ownership is the organisation's decision, best made before launch.

The other side

Subramani is an advocate for the platform she demoed, and she described no production incident. The slow ticket also has a real cost, since the user in her example got nothing.

She said that when she lets audiences try to break the agent, it can fail with a stream of errors. She added that it notices the failure and tries to repair itself. She did not say who is accountable when a self-written fix goes wrong.