- Cloudflare's Data team says it removed about 70 GB/s of backbone traffic in 2026 by sending each event once and copying it late, and by sampling and combining data earlier.
- In one dataset, growth erased about half of a 50% saving within two months, so efficiency buys time rather than a permanent fix.
- Ask which data streams are copied in full, which questions need exact answers, and how quickly growth consumes each saving.
Savings that growth keeps spending
Every infrastructure team knows a win that fades. Cloudflare's Data team put numbers on it in a post published October 13. Each year, it says, it faces "exponential traffic growth against linear hardware growth." Fixing that once does not end the problem.
The lesson here is that efficiency does not buy a solution. It buys time, and growth spends that time. The useful question for a leader is not whether the team optimised. It is how fast the headroom is being used up.
What the pipeline does
Cloudflare's Data team runs the pipeline that carries logs, analytics and observability data from its global network to central data centers. In May it was delivering roughly two billion events per second.
The path has four steps. A service called Logfwdr collects data from sources spread across the network. Central receivers take it in. A buffering system called Buftee holds it. Then the data lands in databases such as ClickHouse.
The team looks back over five years of fixes. By its estimate, they pushed back the need to buy about two and a half years' worth of extra hardware.
Three moves in 2026: shrink the data, then copy it
The principle behind all three moves is to do less to bytes while they travel. Copy them only after they have been made smaller. The first move shows this best.
The first move was to send each event once and copy it late. Before, Logfwdr sent two full copies of most analytics events, one to each central region, so either region could survive a failure. That doubled traffic as more streams adopted it.
Now each event travels once. Replication happens later, from Buftee, after sampling has shrunk the data. Cloudflare reports that the cutover removed about 43 GB/s of transit traffic and added only about 7 GB/s of copying. The net saving was 36 GB/s. Cloudflare credits two causes: copies are made after sampling, and bigger central batches compress better.
The second move was to sample earlier. Sampling means keeping a representative slice of records and scaling the counts back up. Keep 10 of 100 records, and each one stands for 10. Cloudflare already did this at its receivers. It added the same step at the source, grouped by a key each product team chooses.
The effect is uneven by design. Cloudflare says a key with 100 times more data now stores about 4 times the rows, not about 10 times. In its example, 11.6 million events are held as 18,000 rows instead of 79,000. Sparse keys keep their detail. Cloudflare says it checked results by comparing ClickHouse row counts with estimated totals across busy and quiet keys. On three top datasets, it reports at least half the compressed traffic gone, about 25 GB/s or over 2 PB a day.
The third move was to combine records early. Instead of storing every request, the pipeline keeps counts, such as requests per domain per minute by HTTP method. This used to happen at the end, in the inserters. Cloudflare now does it at each stage. On its HTTP dataset, compressed traffic fell from over 9 GB/s to under 2 GB/s, about 80%.
The trade-off is a decision about questions
These moves are not free. Aggregation gives exact answers, but only to questions chosen in advance. Sampling answers many questions, but only approximately. Each requires someone to decide which questions matter before the data is stored.
That is a governance choice, not only an engineering one. Billing and usage figures need exact sums. Exploratory analysis can tolerate estimates. Cloudflare offers its internal teams both options, through products it calls Ready-Analytics and Ready-Aggregates.
How fast headroom disappears
Cloudflare's clearest example is Ready-Analytics. In mid-May, load balancing cut its traffic from about 26 GB/s to about 13 GB/s. Two months later it was back near 20 GB/s. Growth had erased about half the saving.
Sampling then pushed throughput to about 8 GB/s. Cloudflare estimates that without both changes, the dataset alone would have passed 60 GB/s by September. The lighter load also cut resource use on that dataset's receivers by about half. Cloudflare says that freed roughly 750 CPUs and 3.5 TiB of memory.
Questions to put to your team
These are one company's figures at Cloudflare's scale, reported by Cloudflare itself. The pattern still applies to any organisation that pays to move and store telemetry.
First, which data streams do we copy in full for resilience, and does the copy need to be made at the source? Second, which reports need exact numbers, and which would accept a labelled estimate? Third, who chooses the keys and dimensions that sampling and aggregation rely on? Fourth, how quickly does our data volume grow against our budget, and how long will the next saving last?
Savings buy time, and growth spends it. The teams that cope best are the ones that measure how fast.
Produced by the WebPulse Newsroom with AI assistance from the original reporting credited below, and checked against that source by our editorial review. How we use AI.
Original reporting: Cloudflare.
