- In a self-selected VentureBeat survey, 55 of 68 production AI respondents reported GPU use of 50% or less, and 71% of them planned new, added or replacement platforms.
- The survey cannot say why use is low. Finance-signed ROI figures were scarce among both planned buyers and non-buyers.
- Before approving a purchase, ask what the idle chips are waiting for and whether the new platform removes that cause.
Picture a factory running at half capacity. Before building a second one, a sensible owner asks why the first is half empty. That is an analogy, not a finding. But a recent AI infrastructure survey suggests the same question belongs in AI budget meetings.
What the survey found
VentureBeat Intelligence ran its September VB Pulse survey on AI infrastructure. It drew 102 qualified respondents from organisations with 100 or more employees. Of these, 68 ran AI in production and gave a figure for GPU use. GPUs are the specialised chips that run AI workloads.
Fifty-five of the 68 (81%) reported average utilisation of 50% or less. Thirteen reported higher use. Of the 55, 71% plan to bring in a new, added or replacement platform within a year.
The survey has limits. People opted in and reported their own numbers. It does not say why utilisation is low. It cannot show that hardware is wasted, or that any purchase is a mistake.
The findings arrived as Nvidia CEO Jensen Huang urged companies to build their own AI factories. At GTC Berlin on October 21, he said firms should not just be "consuming the AI" but "producing the AI." The survey does not link the two.
What "half used" could mean
Average utilisation is a blunt number. It cannot say where the idle time comes from. VentureBeat's analyst lists reasons a buyer might upgrade despite idle chips. Better job scheduling is one. Smoother links to other systems and cheaper running of an existing workload are others. Support for jobs the current setup handles badly is a fourth.
These possibilities point to different purchases. The survey did not test any of them. They are WebPulse's reading of what the number might hide.
If jobs are scheduled badly, better scheduling may help and more chips will not. If chips wait on slow data paths, the fix lies in integration. If the work is simply small, extra capacity adds cost and no output. If the current setup cannot run a needed workload, a new platform may be justified even with idle chips.
The report offers one example of why the details matter. vLLM's automatic prefix caching reuses work when requests start with the same text. Its documentation says this cuts prompt-processing work, not the time spent generating new words. A speed-up helps only if it matches the workload.
Spending plans and missing proof
In the full group of 102, 63% planned some platform change within a year. Only 10% had an infrastructure ROI figure that finance had reviewed and signed off.
That gap was similar on both sides. Nine in ten planned buyers (91%) lacked such a figure. So did 89% of those with no plans to change. The report calls that difference statistically insignificant.
The analyst concludes that companies are choosing their next platform before finance has validated returns on the current one. The data fits that view. It does not show that the gap drives purchases.
Satisfaction was fairly good. Respondents with a main platform in production averaged 4.0 out of 5 overall, 3.9 for ease of implementation and 3.6 for value for money. One reading is that operational comfort can stand in for financial proof. The survey cannot show whether it does.
Counting the cost of a finished task
The analyst proposes one measure: cost per successfully completed task. It should include inference (running the model), retrieval, storage, repeated attempts and human review. Finance would then compare that cost with the value delivered.
A task needs a quality bar. A support ticket closed with the wrong answer is a failure, not a completion. The cost of repairing a faulty code change belongs to that change.
The report also separates three kinds of memory. Institutional memory is durable organisational knowledge. Agent task memory is the progress an agent carries through a long job. Inference memory is the cached work used while a model serves requests. The report presents this split as the analyst's interpretation, not a survey result.
The survey did cover inference memory. Respondents' top picks for tools they use or are testing were Nvidia at 41% and Dell at 39%. The report says that gap is not significant. A further 16% had not dealt with serving-memory limits or did not know about them. The question mixed use and testing, so it says nothing about deployment.
Leaders and builders weigh different things
Seniority changed the answer. Total cost of ownership, the price of a system across its whole life, mattered to 52% of the vice presidents and directors in the sample. Among individual contributors, only 4% picked it. The report's test finds the gap real, yet the groups are small: 23 vice presidents and directors and 26 individual contributors.
Fitting the existing cloud and data setup was picked most often overall, at 48%. Performance drew 42% and GPU availability 35%. The report says the gap between the first two is not significant. Cost per million tokens rose from 11% in August to 23% in September, though different people answered each month.
What to ask before the next purchase
The analyst advises starting from one agent workflow and finding what limits it. The five steps are these. The product owner defines success and acceptable human help. The data team names authoritative data and access rules. Platform engineers measure latency, utilisation, caching and operating cost. Finance agrees a baseline and checks the benefit. Then the team changes one thing at a time and tracks cost per completed task.
WebPulse adds four questions of its own. What are the idle GPUs waiting for? Would the proposed platform remove that cause, or add capacity to the same bottleneck? What number would show the purchase worked, and who reports it? Who owns the utilisation figure, and what happens if it stays at half?
Half-used chips are not proof of waste. They are a prompt to ask which problem the next purchase solves. Know why the first half is idle before paying for more.
Produced by the WebPulse Newsroom with AI assistance from the original reporting credited below, and checked against that source by our editorial review. How we use AI.
Original reporting: VentureBeat.
