Skip to content
The AI-First Web Talking point

With 25 new models in a quarter, GitHub says testing must never stop

Julia Kasper of GitHub Copilot described a loop of daily and weekly tests run on every new model.

W
WebPulse Newsroom
AI-assisted · 2 min read
Share on X LinkedIn
With 25 new models in a quarter, GitHub says testing must never stop
In brief
  • GitHub Copilot's Julia Kasper said 25 new models launched in the product in one quarter, and her team re-tests each one in its own harness.
  • Kasper said evaluation never stops, but the tests are GitHub's own and run in partnership with the model makers, so they are not a neutral audit.

Julia Kasper, a product manager at Microsoft who works on GitHub Copilot, described a testing routine that never ends. Speaking at GitHub Universe 2026, in a talk on GitHub's channel, she said each new model must be tuned and checked before developers rely on it. The pace is high.

What was said

Kasper said that "we've had in this quarter alone, 25 new models that were launched in GitHub Copilot, which is crazy." She explained that a coding agent has two parts. The model does the reasoning. The harness is the software around it. The harness supplies instructions, exposes tools and runs the loop of requests. She said the same model in a different harness can give a very different quality of result.

Her team's routine starts before launch. GitHub holds weekly check-ins with model makers, adds each new model to its platform, and runs offline tests plus internal use. It shares findings with the makers. Sometimes they send back an updated version of the model.

The tests run on a schedule: regression tests, daily tests and weekly comparisons against other setups. After launch come live experiments. They track user engagement, task success, speed and token use. Kasper closed with the line that "evaluation never stops."

Why it matters

Our reading: a launch announcement is the start of the evidence, not the end. If quality depends on the harness around a model, a vendor's headline score tells a buyer little about their own setup. Teams that build on several models need their own repeatable tests, run on every release.

This also changes the work. Someone has to decide what a passing result looks like. That person effectively defines what working means for everyone downstream. Kasper described GitHub doing this for Copilot and publishing some results on benchmark.github.com. Engineering leaders should ask who writes those tests in their own organisation, and who reviews them.

The other side

GitHub is not a neutral referee. Kasper said Copilot's identity is model choice, and that different vendors keeps "us honest" in comparisons. But she also described contracts, early notice of launches and a win-win exchange with the same companies it tests. The public benchmarks she named show results for Copilot's own harness. They measure one product, not models in the abstract.

In the parts of the talk we reviewed, Kasper did not say who checks GitHub's tests, or how often a launch fails them. Her case is for continuous testing inside one company. Whether independent, vendor-neutral evaluation exists elsewhere did not come up in those excerpts.

Written by the WebPulse Newsroom with AI assistance, and checked by our editorial review: every quotation was verified against the recording's transcript. How we use AI.

The conversation this talking point comes from