- WEKA's field CTO says GPUs can spend up to 60% of their compute recalculating conversation history they have already processed.
- The claim is from a storage vendor and is untested here, but it shows that GPU cost depends on how memory is managed, not only on chip count.
- Ask your team how much GPU time goes to new work, and who keeps the savings if you pay per token.
A GPU that looks busy is not always doing new work. That is the claim behind a VentureBeat interview with Alan McSeveney, field CTO at storage company WEKA. The lesson for anyone paying for AI capacity is that part of the bill may buy memory, not thinking.
What McSeveney says
McSeveney puts a number on the waste. On chips that serve large language models, he estimates, as much as 60% of the processing effort can go to redoing earlier work on conversation history. This serving stage is called inference: the model answering questions after training is finished. He added that in a self-built AI setup, GPUs account for over 80% of the budget.
How the waste happens
In a long conversation, the model keeps working notes on what has been said. These notes are called the KV cache. They sit in the GPU's own memory so the chip does not redo the work with each new message.
That memory is limited. When older notes are pushed out, the GPU may have to rebuild them when the session resumes. Think of a support agent who must re-read a customer's whole case file on every callback. The agent is busy, but the customer's problem has not moved.
Without any plan for managing the cache, McSeveney said, a chip may spend only about 30% of its time on fresh requests. With a sound plan, he said, that figure climbs past 90%.
The fix on offer, and its limits
McSeveney argues that NVMe flash, a fast type of solid-state storage, is a cheaper place to hold this cache than server DRAM, the main memory in a server. He said flash matches or beats DRAM for the job, scales better and costs much less. The idea is to move a paused session's notes off the GPU quickly, let someone else use the chip, and bring the notes back when needed.
WEKA sells a product for this, called Augmented Memory Grid. It keeps the saved notes in one shared pool that any server in the data center can read. A conversation can stop on one machine and carry on from another, with no need to redo the earlier work. McSeveney said this raised the number of users on the same GPUs by 7x compared with offloading to DRAM. Over the same period, he said, the volume of text produced grew from 200,000 tokens to 2 million.
These are the vendor's figures, given in an interview. The source does not describe the workload, the hardware or an independent test. Treat them as a reason to measure, not as a benchmark.
The source also notes that other players are moving the same way, naming Mooncake, recent DeepSeek techniques and Nvidia's CMX context memory platform. That suggests the problem is recognised across the industry. McSeveney adds that cache tuning does not replace tuning the application, workflow and pipeline.
Who keeps the savings
In McSeveney's telling, the gains go to the party running the chips. That covers a firm with its own hardware and a firm that rents bare GPU instances and runs its own software on them. For those paying a model provider per token, his view is blunt: most of the caching benefit goes to the provider, not the customer.
He tied this to supply. As evidence, he cited Google's warning to customers that GPUs it has ordered for its cloud but not yet installed are already spoken for. The same message, he said, told customers without a reservation by December not to rely on on-demand capacity. His conclusion: running some of your own infrastructure "may become a necessity."
That conclusion comes from a vendor who sells infrastructure. It deserves testing against your own costs. The underlying point still holds: efficiency gains are only worth something to the party that controls the hardware.
Questions to put to your team
First, what share of our GPU time goes to new work, and what share goes to rebuilding context? If nobody can answer, that gap is the finding.
Second, where does a paused session's history live, and how long does it take to restore? Third, if we pay per token, does our price reflect the provider's caching savings? Fourth, do we hold capacity reservations, or do we rely on on-demand supply?
The memory of an AI conversation has a price. Know whether you are paying it once or every time the user comes back.
Produced by the WebPulse Newsroom with AI assistance from the original reporting credited below, and checked against that source by our editorial review. How we use AI.
Original reporting: VentureBeat.
