- SenseCore says splitting AI inference into Prefill and Decode stages on different chips raised token throughput about five times in one test. The figure is a company statement.
- The split creates a dependency. SenseCore says in some mixes 30 to 50 Prefill nodes feed one Decode node, so one failure can strand many.
- Buyers of AI capacity should ask providers for output per cost, failover design and test conditions, not only chip counts.
Counting chips has long been the shorthand for AI capacity. SenseTime's SenseCore wants buyers to judge something else: how reliably a facility turns out good-quality output while meeting response-time and cost commitments. A token is the small piece of text a model reads or writes. That shift changes what a buyer should ask for.
What SenseCore described
Xuan Shanming, SenseCore's vice president and CTO, spoke on October 16 at the tenth CSDI Summit in Shenzhen. Pandaily reported the talk. He described a rebuild of the company's inference systems. Inference is the work of running a trained model for users. His argument was that AI agents, which coordinate with one another or work for long stretches unsupervised, consume far more tokens than chatbots do.
SenseCore puts its current volume at roughly 6 trillion tokens daily. Its goal is 10 trillion daily by the last quarter of 2026. It also wants full-year volume to grow about 25 times. Pandaily labels the capacity and speedup numbers as the company's own.
How the split works
Serving a request has two phases. Prefill takes in the user's input and demands heavy computation. Decode produces the answer. It keeps going back to the stored context, so it needs fast memory access.
SenseCore gives each phase different hardware. Domestic accelerators that fit the job focus on Prefill. High-bandwidth GPUs do Decode. The phases hand off through KV Cache links, which carry the saved context. Those links run over a network that mixes 800G, 400G and 200G speeds. All of this runs in mixed-chip clusters at the scale of 10,000 GPUs, at the company's Lingang AIDC site.
Think of a kitchen where several prep cooks feed one cook at the stove. Each person does the job they are best placed for. The stove sets the pace.
The risk the split creates
That picture also shows the weakness. SenseCore says that in some mixes, a single Decode node may be fed by 30 to 50 Prefill nodes built on mid- or lower-tier chips. If that Decode node fails, the Prefill pool behind it can be stranded.
This is what chip-count thinking misses. One large machine fails in one place. A split pipeline can leave many healthy machines idle at once. SenseCore's pitch is that the throughput gain comes from specialisation. The exposure comes from the same design.
The company describes two safeguards. It keeps a few high-end cards inside the Prefill pool as a quick substitute. Nodes can also change roles between Prefill and Decode as load shifts. The report gives no failure-rate or recovery-time data for either measure.
What the five-times figure does and does not show
SenseCore reported roughly fivefold overall token throughput in one test setup after adopting heterogeneous PD scheduling. Its comparison point used an equal number of flagship high-end GPUs. That is a single test, and the company supplied the number.
The report does not describe the workload, the model or the latency targets. It does not say whether the result holds across the platform. Xuan also did not present domestic chips as displacing high-bandwidth GPUs. He described them as gaining a bigger role in Prefill and mixed workloads, while high-bandwidth GPUs keep doing Decode.
The company is also studying a version spread across locations. Prefill would run on mid-tier capacity in a nearby region. Decode would run on high-bandwidth clusters far away. SenseCore's reasoning is that KV Cache for million-token contexts is getting smaller, from nearly 100 GB to about 1 GB on some models. It is also studying a separation of the Attention and FFN parts of Decode. These are research directions, not results.
Questions to put to your team or provider
If you buy AI capacity, ask what you are really paying for. A chip count says little about output. Ask for tokens delivered per unit of cost at a stated response-time target.
Ask how the service behaves when its scarcest component fails. If the provider splits stages across hardware types, ask which stage is the bottleneck. Ask what happens to the other stages when it stops. Ask for the failover design and the evidence behind it.
Ask what conditions sit behind any speedup claim. A fivefold gain in one configuration is a reason to test, not a basis for a contract.
The lesson here is that efficiency and resilience are one question. SenseCore's pitch is that sharing work across specialised parts yields more output from the same hardware. Every shared dependency is also a place where one fault can reach many workloads.
Produced by the WebPulse Newsroom with AI assistance from the original reporting credited below, and checked against that source by our editorial review. How we use AI.
Original reporting: Pandaily.
