- Shopify says its Cyber Chaos Monkey agents ran tens of thousands of offensive engagements against live production systems without an outage or incident.
- A second agent on a different model watches the first in real time and can kill it. Shopify says the run ends if that watcher fails.
- Before letting an agent test production, ask what stops it, how you know that control works, and what credentials and network access the agent holds.
The brake matters more than the engine
Letting an AI agent attack your own live systems sounds reckless. Shopify says it does exactly that. The interesting part is not the attack. It is the brake.
The lesson here is that an autonomous agent is only as trustworthy as the control that can stop it. Shopify Engineering published a description of its system on November 12. It calls the system Cyber Chaos Monkey, or CCM. The name nods to Netflix's Chaos Monkey, a 2011 tool that randomly shut down production servers to test whether services could survive real failures.
What Shopify says it built
Shopify says it gives an agent live credentials and everything it knows about a system, including bugs, infrastructure and source code. The agent then looks for new ways to bypass Shopify's safeguards.
The agents send live requests to Shopify's production infrastructure, using live test shops and organizations. Shopify frames this as the way to keep merchant data and availability out of harm's way. It likens CCM to a bug-bounty researcher with access to source code and past reports.
Shopify reports that its agents have run tens of thousands of offensive engagements. It says they produced dozens of impactful findings and caused no outage or incident. These are Shopify's own figures. The post does not say what the findings were or how their impact was judged.
Shopify also gives a reason for the effort. It says reports to its bug bounty program are up almost 300% since 2024. It links this to frontier models pushing the cost of finding vulnerabilities toward zero. It says sorting through that volume makes the program much harder to run.
How the system works
Shopify splits CCM into two parts, Discovery and Sandbox. Discovery takes an idea from a human security engineer, another security agent or its own feedback loops. It then uses the latest cyber models to refine that idea and spot code or settings that could lead to a weakness.
Promising leads go to the Sandbox, where agents try to exploit the bug. They install packages, build fuzzers and send live requests to Shopify. They then read production logs to see how each request was handled and compare that with the source code. Findings flow back to Discovery, which can start further engagements.
Two keys, two different models
Shopify gives every agent Rules of Engagement. The agent may try whatever exploit it likes. The restrictions begin once something works: it may not pivot to other systems or hold on to access.
Shopify says it does not trust the agent to stay within those rules. So it borrows two-person integrity, a practice it says governments use to protect sensitive information and that is also used in risky red team operations. A second party must review the first party's actions.
In CCM, that second party is a "judge" agent running on a different model. It reads every event from the "executor" agent in real time, including its thinking and tool calls. If it sees behaviour that breaks the rules, it can kill the executor at once.
Shopify describes the design as failing closed. It says that if the judge times out, crashes or cannot assess messages, the run ends. Both agents run beside a controller in the same Kubernetes pod, a unit of containers that runs together. The controller exits if either agent stops responding, and the pod is not allowed to restart it.
Containment around the agents
Shopify describes three further controls. The executor runs with no escalated permissions. It holds only short-lived credentials that it can read but not change.
When the agent calls Shopify's knowledge services or model providers, a credential proxy swaps a placeholder in its requests for a real credential. Shopify says it strictly controls which headers go to which domains, and the credentials rotate. An egress proxy then allows traffic only to Shopify production domains, its knowledge services, model providers and a few package managers.
Shopify also points to recent cases in which AI agents doing security testing ended up in systems they were not meant to reach. In its view, that is why limits have to be part of the design from day one.
The bet and its limits
Shopify calls CCM a bet on asymmetry. Models help every attacker, it argues, but Shopify knows its own systems better than any outsider. It hopes to find issues first and make it costly for attackers to find what it misses.
That is a hypothesis. The post gives no measure of attacker cost, and it cites no independent verification of the results. It is one company's account.
The post also reports no judge interventions, false negatives or false positives. "No outage or incident" fits a judge that works. It fits equally well a judge that was never tested. The brake is the heart of the design, yet the post offers no evidence of how reliably it catches rule-breaking.
What to ask your team
If anyone proposes letting an AI agent test production, start with the stopping mechanism, not the agent's skill. Ask these questions.
What can halt the agent mid-run? Does that control fail closed, so the run ends if the controller breaks? Is the watcher a different model from the actor? How do you know the watcher actually catches violations? What credentials does the agent hold, and can it alter them? Where can it send traffic? And who will triage the extra reports that faster discovery brings?
Autonomy is easy to buy. A dependable off switch, with proof that it works, is what to ask for.
Produced by the WebPulse Newsroom with AI assistance from the original reporting credited below, and checked against that source by our editorial review. How we use AI.
Original reporting: Shopify Engineering.
