- Huawei told the ASC 2026 conference that moving an AI model's working memory, the KV Cache, between compute, storage and network is a growing limit on inference.
- Huawei says its lab tests showed 7.3 times the inference throughput from KV Cache offloading, but its announcement left out the full test setup.
- Buyers of AI capacity should ask vendors how memory traffic is handled and ask for tests on their own workloads.
An AI agent on a long task must keep track of everything it has read and done. Where that memory sits, and how fast it moves, now shapes what agents cost to run. At a storage conference in Shanghai, Huawei argued that this movement has become a limit on speed.
What Huawei presented
The talk came on October 22 at the 2026 Advanced Storage Conference, organized by the Open Data Center Committee. Pandaily reported it. Huawei's subject was how to link storage and networks for new AI workloads.
This was not a product launch. Huawei introduced Stellar AI Fabric and its Hyper-Converged Fabric (HCF) design at HUAWEI CONNECT 2026 in September. Pandaily says the Shanghai talk adds the first KV Cache figures for that design.
Why a cache becomes a problem
The KV Cache stores partial results from a model's attention step. Reusing them saves the model from reworking every earlier token each time it writes the next one. Think of a clerk who keeps notes on a case file instead of rereading every page for each question.
The notes pile up as the case grows. Huawei points to agents that handle very long inputs and many back-and-forth turns. In its account, three things feed each other: the size of each request, the cache held in memory, and the traffic between nodes.
The result, Huawei says, is a cache across a whole inference cluster that nears the petabyte scale. It adds that sudden surges of cache data moving between nodes strain the data center network.
Three walls, as Huawei describes them
Huawei names three obstacles. The first is compute. Accelerator cores are weak at input and output work, and that work uses compute the model needs.
The second is delay. Data passes between layers and is converted again and again between protocols. Long-tail delays make it worse. These are the occasional slow network responses that can hold up a whole request.
The third is IOPS, the number of reads and writes a storage system completes each second. Huawei says older software stacks keep the storage behind them from running at the speed it was built for.
The proposed fix and the numbers
Huawei's plan has three layers. At the accelerator, it wants processors to request cache data directly. In storage, it moves operations close to the drives and uses vector acceleration there.
On the network, HCF schedules traffic with quality-of-service rules tuned to cache data, coordinated from end to end. Huawei credits the network layer alone with a storage IO gain above 30%.
The big number comes from Huawei's own lab. It used the GLM-5.2 model. Huawei says sending the cache straight to storage produced 7.3 times the tokens per second of a standard memory-based setup. Tokens per second measures how fast the model produces text.
Huawei adds that coordinating storage and network scheduling brought a further 20% gain in inference efficiency.
These are vendor figures. The announcement leaves out the complete test setup. Without the baseline, context length and load, a buyer cannot tell how far 7.3x would carry to another system.
The idea behind the news
This shows that the cost of an agent is partly the cost of its memory. Much of the AI spending debate centers on accelerators. Huawei's talk points to the plumbing around them. That means the storage, the network and the software that move remembered context.
Two questions are left open. The design spans accelerator, storage and network layers. A buyer needs to know how much of it depends on one vendor's hardware.
The second question is about privacy. If conversation context sits in shared storage, who can read it? How is it protected? The source does not address either point.
What to ask your team and vendors
Ask any vendor that quotes an inference speedup for the baseline, the model, the context length and the number of simultaneous users. Ask whether the gain holds on your own workload. Ask whether it is measured in total throughput or in response time for one user.
Ask your infrastructure team where cache data lives today. Ask how much network traffic it creates and what access controls cover it. Ask which parts of any proposed design work with other vendors' equipment.
The lesson is that an AI system is only as fast as its slowest hand-off. Before buying more compute, find out what it spends its time waiting for.
Produced by the WebPulse Newsroom with AI assistance from the original reporting credited below, and checked against that source by our editorial review. How we use AI.
Original reporting: Pandaily.
