Maersk engineer: AI refactors can drop branches quietly, so green tests prove little

Dmitry Buykin, who works at Maersk, said on the AI Engineer show that a passing test run is weak proof an AI-rewritten change is safe. He drew on his own large codebase, where agents lost logic and skipped work without any error. He now leans on replaying frozen real tasks and on a second agent as checker.

What was said

Buykin said he used to trust green. A refactor is a rewrite meant to leave behaviour unchanged. His warning: "AI refactors fail quietly."

His example was a decision table, a list of rules that decides what happens to an order. Each rule is a branch, a path the program can take. After a rewrite, each must still exist exactly once. One branch, for amounts over the limit, was lost. An order would have been approved with no team lead involved.

His check is a fixed set of real past tasks. He runs the old version and the new one on them and compares. A branch that no real task reaches counts, in his words, as no evidence.

He also said a single agent cannot judge its own work. A second agent reviews it against a checklist, never seeing the author's reasoning, so "It can't be talked into agreeing." Over six months, he said, this became most of his checking. He added that agents can leave part of a batch undone without saying so. In one backlog, most items looked covered, but only about a quarter could really be closed.

Why it matters

Our reading starts with a mechanism Buykin did not spell out. A test suite only checks behaviour someone wrote a test for. If a rewrite drops a branch no test reaches, nothing fails and the suite stays green.

That is how AI multiplies an old weakness. Agents change code faster than people can review it, and the signals managers rely on, such as tests and dashboards, stay quiet. Buykin himself said verification speed is the limit.

For anyone buying or running AI-written software, we would swap "do the tests pass?" for questions like these:

Which real tasks did you replay, and which branches did they reach? Who checked the work, and did that reviewer see the author's reasoning? What was skipped, and how do you know?

The other side

This is one practitioner describing one codebase, one lost branch and one backlog. It is his experience, not a measured industry rate.

His method has limits. A branch that no real task reaches is only missing evidence, not proof that the branch works. Checking also takes time and attention.

The second agent still relies on a checklist that someone must write and keep current. Buykin also said that for the prose instructions in his system, only people can judge what must survive a rewrite. The talk did not show that a second agent catches everything the first one misses.