- Carl Sverre argued that an LLM agent does not change how to think about testing. It widens the range of inputs that the surrounding code must handle.
- Tests that only confirm expected behavior cannot cover that wider range. He pointed to deterministic simulation testing, a method his company sells, as one answer.
Adding an LLM agent to an application does not create a new testing problem. It makes an old one bigger. Carl Sverre, who works on the Red Team at Antithesis and leads much of its AI strategy, made this point on Software Engineering Daily. The agent widens what the rest of the code may receive. Tests that only confirm expected behavior therefore fall short.
What was said
Host Sean Falconer asked whether software that depends on LLM output changes how teams think about testing. He noted that teams can run evals, which measure how often a model's answer takes the expected form. Sverre disagreed that the thinking changes. "I don't think it changes it," he said. Then he added: "I think that it opens up the state space that you need to test quite a bit." The state space is the full set of situations a program can end up in.
His example: an application asks the LLM what to do next in a multi-step process. The possible next steps can multiply. In the extreme case, the agent runs in a Linux sandbox and can do almost anything on the machine. Then all the software reacting to the agent must cope with a much wider range of inputs.
Sverre also said most tests lock in features a team has already built. In his words, "it's actually really hard to write tests that test for the unknown, unknown part of the space." For this gap he pointed to deterministic simulation testing. It runs software in a controlled setup where randomness, time and event order can be replayed exactly, so many possible paths can be explored.
Why it matters
Our reading: the risk sits less in the model's answer than in the code that acts on it. An eval tells you how often the model answers in the expected form. It does not tell you what the surrounding software does when the answer is something else. Teams building agent features could ask whether their tests reach past the paths someone wrote by hand. Buyers could ask vendors the same question. Sverre's framing suggests the extra test effort belongs around the agent, not only on the model.
The other side
Sverre runs a company that sells this kind of testing, so he has a stake in the argument. His vivid example was an "extreme case", an agent free to do anything in a sandbox. The excerpts do not say how wide the range of agent actions is in typical products. Many applications may limit what an agent can do, and that would narrow the problem. The excerpts also offer no case of an agent causing a real failure.
Cost is another open point. He said Antithesis makes software deterministic automatically for Linux software. Otherwise a team must rewrite its code to control time, randomness and scheduling. That is real work, and the conversation did not size it.
Written by the WebPulse Newsroom with AI assistance, and checked by our editorial review: every quotation was verified against the recording's transcript. How we use AI.
The conversation this talking point comes from
- Software Engineering Daily: Software Verification in the Age of Coding Agents (2026-10-15)
