- Steven Latré of imec said three coding harnesses reached similar accuracy, yet for a GLM model one used half the GPU time.
- Our reading: if this holds beyond one test, teams running their own GPUs should compare harnesses on GPU time per task, not accuracy alone.
Steven Latré runs the AI effort at imec, a Belgian research center that works on chips. On the AI Engineer show, he argued that once a company pays for GPU time, the harness affects cost, not only the model. A harness is the software that wraps a model so it can work as a coding agent. In his team's test, three harnesses gave similar results, yet one used half the GPU time for a GLM model.
What was said
Latré's team tested open-weight models, which anyone can download and run, with three coding harnesses on a benchmark called SWE-Bench Pro. Accuracy was roughly the same across harnesses and models.
At first glance, he said, that suggests the harness does not matter, because the job gets done. He argued that may hold for API-only users. On your own GPUs, the time the hardware spends working becomes the cost.
In his data, fewer tokens meant less working time. Tokens are the small chunks of text a model produces. For a GLM model, one harness generated half the tokens, so it used half the GPU time. Latré said: "the GPU bill you would in the end have would be half of the other harnesses as well."
Why it matters
Our reading: accuracy scores alone hid this cost. In Latré's test, a team reading only accuracy would have missed a twofold difference in GPU time.
If this holds beyond one test, someone must measure tokens and GPU time per finished task and compare harnesses on that basis. That moves effort from building agents toward judging them.
It would also strengthen the case for training people to run fair side-by-side comparisons. That is an inference, not something Latré said.
The other side
Latré gave limits himself. His results cover one application, coding assistance, and a real company runs a mix, which he said complicates things further. He called the links between harnesses, models and hardware very complex.
He did not say why one harness generated fewer tokens, and we cannot supply a cause. That is the question to put to vendors. His team also had not configured details such as quantization, which shrinks a model, or speculative decoding, a speed-up technique. The field keeps moving, too.
His team plans to publish a public benchmarking data set and launch an alliance, saying it cannot do this alone.
Written by the WebPulse Newsroom with AI assistance, and checked by our editorial review: every quotation was verified against the recording's transcript. How we use AI.
The conversation this talking point comes from
- AI Engineer: When Running Your Own AI Models Beats Paying for APIs - Steven Latré, imec (2026-10-31)
