- Steven Latré of imec said his team's tests put the break-even GPU usage at 89% on small systems and about 15% on large ones.
- With GLM, one of three coding tools used half the tokens of the others at similar accuracy, which Latré said would halve the GPU bill.
Steven Latré leads AI at imec, the Belgian chip research centre. On the AI Engineer podcast he said that moving from paid APIs to your own GPUs is not a simple choice with a straight-line payoff. GPUs are the chips that run AI models. His team's coding-agent tests put break-even at 89% busy time on small systems and about 15% on large ones.
What was said
Latré's team ran thousands of benchmark runs on coding agents with open-weight models. They looked at total cost of ownership, which counts everything it costs to run the system. He said hardware cost does not follow a smooth line. It is often "just a stepwise function", with jumps as you move up in hardware.
The break-even point depends on the size of the system. At the lower tiers of models and hardware, Latré said, "your GPU needs to run at capacity at 89% of the time." On big B200-class machines, he said, 15% utilisation is enough, and that is realistic.
For GLM-class models, he said, the full analysis for coding assistance showed a cost saving of around 25%.
He also compared three coding tools, which he called harnesses. They reached about the same accuracy. But with GLM, one produced only half the tokens, the units of text a model generates. GPU time fell by half too, so its GPU bill would be half that of the others.
Why it matters
Our reading: the saving is a forecast, not a fact. An API charges for what you use. Owned hardware charges whether or not anyone uses it. If a buyer assumes high demand and demand falls short, the buyer carries the cost of idle chips.
For managers, the question before approving a GPU purchase is simple. What utilisation does the business case need, and what evidence says you will reach it? Latré's figures show the answer differs sharply by system size.
The harness result adds a second check. Accuracy scores alone hid a twofold gap in cost. With an API only, Latré said, that gap may not matter to you. On your own GPUs it shows up in the bill.
The other side
Latré was clear about the scope. He showed results for one application, coding assistance, so other workloads may behave differently. He also called this the first study his team has done on the topic in recent months.
He said cost is only one reason to switch. His team looked at it purely as total cost of ownership. Other reasons, which he did not detail, could justify owning hardware.
A saving of about 25% is modest. The excerpts do not say how often real teams reach the needed utilisation, or what happens when they miss it. The idea that buyers must guess demand in advance is our inference, not something Latré tested.
Written by the WebPulse Newsroom with AI assistance, and checked by our editorial review: every quotation was verified against the recording's transcript. How we use AI.
The conversation this talking point comes from
- AI Engineer: When Running Your Own AI Models Beats Paying for APIs - Steven Latré, imec (2026-10-31)
