- Julia Kasper said GitHub Copilot tests changes offline, usually ships them to staff, then uses public metrics to decide on full rollout.
- She said evaluation never stops, and that enterprise customers now look at token efficiency and cost per user.
GitHub Copilot's coding agent keeps changing, because the team retunes it for each new AI model. Julia Kasper, a Microsoft product manager, made the case in a GitHub Universe 2026 session. It was published on GitHub's YouTube channel as "How we optimize GitHub Copilot for every new model."
What was said
Kasper began with the pace of change. She said "we've had in this quarter alone, 25 new models that were launched in GitHub Copilot, which is crazy."
Copilot's answer is a harness. That is the software wrapped around a model, which shapes how it works. Kasper said the same model in a different harness can produce very different quality. Models also differ in habits. She joked that some may be more or less lazy about calling a tool, meaning asking a helper program to do a step.
Each change starts as a hypothesis, she explained. One example is a change that uses fewer tokens, the units AI models charge by. Offline tests run when the code is merged, to check for regressions.
Once the team is confident, an online experiment begins, usually with staff shipping. That yields some internal data first. The team then wants the change public as soon as possible. Public metrics help decide whether it is fully rolled out.
Those metrics include user engagement, task success, latency and token use. A sudden drop in users switching away from a model is one signal. Kasper said more enterprise customers, Microsoft among them, now look at token efficiency and cost per user. Her summary: "evaluation never stops."
Why it matters
Our reading: because the harness keeps changing, a buyer is judging a moving target. A pilot that went well in spring may not describe the tool in autumn.
Kasper's own scorecard is a template. Track task success, latency and cost per user over time, not once at purchase.
Three questions follow. Who sees a harness change before it ships? How would we learn that behaviour changed? How is cost per user reported to us?
Our reading also notes that public signals, such as users abandoning a model, arrive after release. Ordinary customers are part of the safety net. Ask what protects them when a change goes wrong.
The other side
This is one vendor describing its own process, and it is a careful one. Staff are real Copilot users, so internal data is not a lab result. The show notes also say offline evaluations catch regressions before developers do.
In the excerpts we reviewed, Kasper gave no figures on how often a change is pulled back after public release. She did not say how customers are told when behaviour changes. The machine transcript is also rough in places.
Written by the WebPulse Newsroom with AI assistance, and checked by our editorial review: every quotation was verified against the recording's transcript. How we use AI.
The conversation this talking point comes from
- GitHub: How we optimize GitHub Copilot for every new model (2026-11-18)





