As AI writes the code, knowing what good means becomes the scarce skill

Laurie Voss, co-founder of npm and head of developer relations at Arize AI, made a point about skill on the AI Engineer show. As agents take over the typing, he argued, the scarce ability is saying what "good" means. In his telling, that knowledge sits with the people who run the business or maintain the code. It does not sit in the model.

What was said

Voss pointed to a benchmark from Cognition, the maker of the coding agent Devin. It did not ask whether code passes tests. It asked whether the maintainer of a repository would merge the change. On that question, Voss said, the pass rate fell from 87% to 54%. He blamed things a test cannot see: style, side effects, outside systems and history held only in the maintainer's memory. He called the missing piece "the sense of what good means."

His other example was a bakery owner who needs to track orders. Many programs do that, he said, but few suit a bakery, and an auto parts store needs something else. That knowledge, he said, never reached the training data or the model weights. It lives with the owner, and the builder must ask for it directly.

He also explained why faster typing has not meant faster delivery. In his words: "the cost of producing plausible code has collapsed while the cost of knowing whether to trust that code has stayed exactly the same." Code got about eight times faster, he said, yet shipping barely moved. The show notes put the gain at about 30%. Review is the first constraint, he said, with operations and security next.

Why it matters

Our reading: tests measure whether code runs, while a maintainer measures whether it belongs. That gap is where the cost of agent-written code now sits. If a buyer pays for cheap code, the bill moves to review and judgment.

Three questions follow for anyone signing a budget. Who in your organisation holds the unwritten merge criteria? Is any of that written down, or does it leave with that person? What does review cost you now that code is cheap?

Voss added that this work does not transfer between products. A calendar app and a scheduling app need different definitions of good. He said no one trains engineers to spend a week with a baker and return with a spec. He called the position of junior engineers a failure the profession must fix. The excerpts do not say how.

The other side

Voss admitted he may be falling into a trap common in AI circles: claiming everything can be automated except the thing he is good at. He heard a venture capitalist make that same claim and told him that someone would soon automate his job too.

He also said defining good can be written down. Cursor's review tool Bugbot turns every thumbs-down from a team into a rule, and he said it now has about 44,000. Once a maintainer puts knowledge into writing a model can train on, he said, the bottleneck moves again. His view that judgment stays scarce for ten years is, in his words, a bet.