Duolingo's agents push legacy files down the test queue, so old code gets less help

Duolingo's test-writing agents work on modern code first and push legacy files down their queue. Kush Agrawal of Duolingo described this in a talk on the AI Engineer show.

What was said

Agrawal said Duolingo's iOS app grew from about 200,000 lines of code in 2019 to close to 2 million. About half the code follows MVVM, a pattern that splits an app into layers that are easier to test. The other half is legacy code the team is still migrating. The project began with the MVVM half because it is the easiest to test.

A selector script gives each Swift file a score for how likely an agent is to write a good test for it. Agrawal said view models, repositories and data sources get a 2.5x boost. Legacy files are pushed down, and large files are penalized.

He also described one fix. A giant user-client singleton, meaning one shared object, was passed around in two forms. Some code used the concrete class, which is hard to mock, meaning hard to swap for a fake in tests. Other code used a protocol, an abstract description of the class. The team switched everything to the protocol and added a lint rule to require it. Agrawal said this made many files easier to test, for agents and for humans.

Coverage in the MVVM slice rose from 9% to 59%, and overall coverage from 11% to 24%. Agrawal said the same loop now applies to modernization: "Unit tests were just the lowest hanging fruit, and any job that fits this shape can run on the same infrastructure."

Why it matters

Our reading: the selector score works like a price list for legacy, though Duolingo did not price it. Old files score lower, so they get less automated help.

The singleton example shows the other direction. One shared cleanup made many files cheaper to work on. For leaders who budget for modernization, that is a way to size the work. Ask which parts of the codebase an agent can safely change today, and which single dependencies hold the rest back.

The other side

The score reflects the team's confidence in writing tests. It is not a measured cost of legacy code. Duolingo gave no coverage or cost figure for the legacy half alone.

The modernization work is early. Agrawal said Duolingo is starting to apply the loop there, and he gave no results.

Agrawal also warned that "agents sometimes take the shortest path to green CI." A passing test is therefore not proof of a good one.