- A speaker on AI Engineer argued that platforms hosting arbitrary agents cannot predict what those agents will ask for, so fixed tests fall short.
- His team had an agent write 1,000 varied tests against their own sandbox, and the failures showed up at both the interface and the sandbox layer.
Software that hosts arbitrary AI agents cannot be tested with a fixed checklist, because nobody knows in advance what the agent will ask for. That was the argument in a talk on the AI Engineer show, billed as coming from Kevin Orellana, an engineer on Amazon Bedrock AgentCore at AWS. His answer is to let an agent generate a wide, messy set of tasks and see what breaks.
What was said
The team wrapped their cloud sandbox and cloud browser in a server. A sandbox is a locked-down computer where agent code runs. They then asked a coding agent to write and run 1,000 tests across 10 domains, and logged the results in a SQLite database.
A speaker on the show described the number as loose: "1,000 tests here was pretty arbitrary. It was just to cast a white net and see what caught back."
The net caught real faults. An uploaded image did not come back identical. Null bytes, a special character that can mark the end of a string, threw off the tests. Files written to one directory could not be retrieved. Browser tests with many tabs ran out of memory.
The speaker contrasted this with a classic API service, where you send a request and roughly know what the backend does. An agent in a sandbox may download a surprise package or convert files for no obvious reason. His advice: "sure, test it locally, but also fuzzy test it with a wide set of domain tests remotely."
Why it matters
Our reading: the talk shows testing work shifting from writing cases to judging results. The speaker said he doubts anyone still writes unit tests by hand. The harder job is looking at 1,000 outcomes and asking where each failure comes from.
His team split the faults into two layers: the interface that exposes the sandbox, and limits in the sandbox itself, such as memory and file permissions. That sorting is judgment work. For buyers, it means a platform that passes its own scripted tests may still fail on the workloads your agents invent.
The other side
This is one team testing its own product, and the speaker called the 1,000-test count arbitrary. The talk does not claim that number gives full coverage, and it gives no rule for when fuzzing has done enough.
It also leaves open how people should train to judge such results at scale. The speaker said his team let the agent work out whether a failure was a service issue or an interface issue. How much of that sorting a human must check was not settled.
Written by the WebPulse Newsroom with AI assistance, and checked by our editorial review: every quotation was verified against the recording's transcript. How we use AI.
The conversation this talking point comes from
- AI Engineer: 1,000 Agent Tasks in a Sandbox: What Breaks When LLMs Run Code — Kevin Orellana, AWS (2026-10-12)
