- Julia Kasper of GitHub Copilot argued that the same model can give very different results depending on its harness, the system of prompts, tools and loops around it.
- Her team treats each new model as a tuning job, checked with offline tests and live metrics, and she said that evaluation never stops.
A coding agent is more than the AI model inside it. Julia Kasper, a product manager at Microsoft, made that point at GitHub Universe 2026, in a session published on the GitHub channel. She argued that the layer around the model, called the harness, heavily influences how good the results are. Her team spends its time tuning that layer for every new model.
What was said
Kasper described an agent such as GitHub Copilot in VS Code as two parts: a model and a harness. The model is the reasoning engine. She said her team has little influence over it.
The harness is everything around the model. It supplies the system prompt, which gives the model its instructions. It exposes the tools and their schemas, the formats that tell a model how to call each tool. It also runs the agent loop, meaning how a session proceeds and how tools are called.
Her conclusion was direct. "So the same model, a different harness could essentially heavily influence the quality of an outcome."
The pace makes this a constant job. Kasper said that in one quarter alone, 25 new models launched in GitHub Copilot. Each can behave a little differently, especially across vendors. Her team checks every new model with offline evaluations before release, then watches production metrics as it rolls out. She also said models can be "a little bit more or less lazy to call a certain tool." Her summary: "The harness is the product."
Why it matters
Our reading is that this moves the skilled work in coding agents. Building a model is not the job of most software teams. Judging how a model behaves inside your own setup, and adjusting prompts and tool definitions until it behaves well, is.
For managers who buy these tools, the lesson is to ask how a vendor tests. Kasper said evaluation never stops, even after a model launches. A vendor that ships new models without that loop may deliver uneven quality.
It also suggests a training need. Teams who customise agents need people who can design tests, read results and spot a regression, not only people who write code.
The other side
This is one product team describing its own work, and the session sells a view that the harness matters. The excerpts give no numbers showing how much quality moves when only the harness changes.
Kasper also said the model is where knowledge and capability come from. Her team works hand in hand with model providers, and holds weekly check-ins with them. So the harness does not replace the model. It sits beside it.
The excerpts do not say what skills this tuning work needs, or how teams outside a large company could afford the same testing.
Written by the WebPulse Newsroom with AI assistance, and checked by our editorial review: every quotation was verified against the recording's transcript. How we use AI.
The conversation this talking point comes from
- GitHub: How we optimize GitHub Copilot for every new model (2026-11-18)





