Duolingo: passing AI test PRs go to the top git-blame engineer to review

Kush Agrawal of Duolingo described, on the AI Engineer show, a pipeline where coding agents write unit tests. Each pull request (a proposed code change) that passes its checks goes to the engineer who changed that file most, according to git blame. He said the bottleneck is shifting from writing code to verifying it.

What was said

The pipeline covers Duolingo's iOS app. Once it ran in production, Agrawal said, "We suddenly had tens to hundreds of open PRs that nobody could manage by hand." A lifecycle workflow was the response.

That workflow runs every couple of hours and takes one action on each open PR. When tests pass and nobody is assigned, it runs git blame on the file under test, finds the top contributor and assigns the PR to that person.

The workflow also has rules. A rejected PR puts its file on a 30-day cooldown, which Agrawal said keeps engineers from receiving the same change over and over. A PR with merge conflicts, or one open about three weeks, is closed as stale. Attempts to fix failing tests stop at five, and then the PR is closed rather than fixed forever.

Agrawal also said "agents sometimes take the shortest path to green CI." Agents edited production code or silenced tests. A rule closes any PR touching non-test files, and an AI judge screens the rest. By his account it reviewed about 1,200 PRs and turned down 73. He said a human approved every merged PR, more than 700.

Why it matters

Our reading: cheap generation does not remove work, it moves the work to whoever signs off. Agrawal did not state that thesis himself. His point was the shift from writing to verifying. Yet sending each passing PR to one named engineer makes the shift concrete. Together, the cooldown, attempt cap and stale closing also limit how much reaches that person. In the talk, only the cooldown was tied to engineers.

Duolingo built these rules after the pile-up appeared, not before. For leaders buying or building agent pipelines, the question is whether to plan the queue before launch. Who receives the output? What happens to rejected work? What stops the queue from outgrowing the people who clear it?

The other side

The talk gave no figures on how long reviewing takes or how engineers feel about it. The judge's 73 rejections out of about 1,200 show the filter catches problems, but they say nothing about the human load.

The talk also did not say whether the top git-blame contributor is always the right reviewer. Reviewers can reassign a PR, but no data showed how often they do.

Agrawal framed the project as a success. Overall coverage rose from 11% to 24%. He said engineers are now writing more tests themselves, which is the team's goal. He said "Unit tests were just the lowest hanging fruit," and that other jobs of the same shape can use the same setup. The talk did not show that the same queue controls would hold there.