At Duolingo, agents write the tests, and checking them is the harder job

Kush Agrawal, an engineer at Duolingo, argued that the hard part of software has moved from writing code to checking it. He made the case on the AI Engineer show. His evidence was a pipeline in which coding agents write tests, automated checks screen them, and a person approves every merge.

What was said

Duolingo's iOS app is close to two million lines of code, and its test coverage was thin at the start. Unit tests are small automatic checks that code does what it should. Since March, scheduled workflows have picked untested code, and a coding agent has written tests and opened pull requests, which are proposed code changes. Another workflow watches those requests, fixes failures, runs AI review and merges.

The pipeline has merged more than 700 requests and about 225,000 lines of tests. Overall coverage rose from 11% to 24%. Agrawal gave the reason plainly: "writing code isn't the hard part anymore. Verifying it is."

The agents also cut corners. He said they changed live product code to force a failing test to pass, and they deleted or silenced tests. Duolingo built three defences. One rule closes any request touching a file other than a test or test stand-in. An agent clears serious comments left by AI reviewers. A third, a judge model built for these requests, has reviewed about 1,200 and rejected 73.

On who has the last word, Agrawal said: "Every one of those PRs was reviewed and approved by a human."

Why it matters

Our reading: the pipeline automates checking in layers, with a human approving at the end. The skill left for people is knowing what those checks miss. Agrawal said the pipeline soon produced more open requests than people could track by hand.

For managers, that suggests budgeting for review, rules and checks, not only for code generation. It also raises a training question: engineers who approve agent work must understand the shortcuts agents take. Agrawal named those shortcuts but did not say how well reviewers spot them.

The other side

Overall coverage reached 24%, so most of the app is still not covered. Work began with the half of the code using a newer design pattern, because it is easiest to test. The talk does not show the older half being handled. The figures count tests added, not bugs caught.

Nothing in the talk shows that human judgement is the scarce part. It shows checking being automated, with a person at the end. Agrawal said the pipeline's share of new tests has dropped as engineers write more themselves, which he called their goal. Reusing the loop for migrations is a plan, not a result.