- Transformers, now generally available in Cloudflare Logpush, lets teams filter, redact and reshape logs with a SQL query before delivery to a SIEM or storage.
- The design fails closed. If a transform cannot run, Logpush holds the batch instead of sending raw data, so teams must watch for delivery stalls.
- Leaders should ask which log fields reach their destinations, who owns the rules, and who is alerted when a transform fails.
Many teams run a separate cleanup service after their logs are collected. Cloudflare's Logpush feature suggests another order: decide what a log may contain before it is delivered. That is a small change in the pipeline and a large change in who controls the data.
What Cloudflare released
In an October 12 engineering post, Cloudflare detailed Transformers. The feature became generally available during Cloudflare's recent Birthday Week, when Logpush also opened to all self-serve plans. Logpush exports Cloudflare logs to security monitoring systems (SIEMs), analytics platforms and object storage. Transformers is available on Free, Pro, Business and Enterprise plans.
A Transformer is a SQL query tied to one Logpush dataset. Records that fail its WHERE clause are removed. The SELECT statement shapes each record that remains. Teams can add, rename, remove or nest fields. The output is NDJSON, meaning one JSON record per line.
Cloudflare's worked example masks part of the client IP address. It also tags records with a provider name and drops internal and health-check requests. Customers build and preview queries in Transformer Studio or through the API. Each save creates a new version, so changes can be reviewed.
The idea: the export step is a control point
Cloudflare describes today's common approach plainly. Teams often need a separate processing service to clean logs, and they must deploy, scale, monitor and maintain it. Cloudflare adds that teams also pay to export, ingest and store records they discard the moment they arrive.
The lesson here is that the moment before delivery is a cheap place to enforce a data rule. Think of a mail room that blacks out pages before anything goes into the courier's bag. A rule applied downstream protects only what comes after it. Every system in between has already seen the raw record.
One boundary matters. The transform runs inside Cloudflare's own delivery pipeline. It controls what Logpush delivers to your SIEM or storage. It does not change what Cloudflare itself generates or holds.
How it works under the hood
The engineering choices explain why this is credible at scale. Cloudflare's first prototype ran customer code in Cloudflare Workers. That put a network call into the export path, which could add timeouts, retries and delay.
Cloudflare instead runs the SQL inside its delivery service, called Bufstream. That service is written in Go. It uses Apache DataFusion, a query engine written in Rust, and passes data between the two as Apache Arrow batches, a typed, column-based format. Cloudflare says this avoids parsing each record one by one.
Each Logpush dataset has a generated schema, so a query is checked against known fields and types. For each time slice of a job's data, Bufstream prepares the query once as a plan. It then reuses that plan on every batch in the slice.
Two caveats apply. The batch was synthetic, and Cloudflare ran the test itself. The post gives no figures for real customer workloads.
Fail closed, and what that costs
The most important design choice is what happens on failure. If a transform cannot run, Bufstream does not advance the job's delivery offset and does not send the untransformed batch. Cloudflare's stated reason is that raw data could break the redaction or filtering assumptions of downstream systems.
That protects the privacy rule. It also means a broken query can delay logs. Bufstream retries with growing delays and reports errors on the job. If a job keeps failing for a long period, Cloudflare disables it without human action.
For a security team, a disabled job means logs stop reaching the destination. The post does not say how long held data is retained. Cloudflare provides a Transformer Health view showing successful and failed transforms, data volume before and after, and duration. Someone has to watch it.
Questions to put to your team
Start with the data itself. Which fields in our logs are personal or sensitive, and do all of them reach the SIEM today? Redacting before delivery can shrink that exposure, but only if someone writes the rule.
Then ask about ownership. Who writes and approves the SQL, and who reviews version changes? A redaction query is now a control, and it needs an owner like any other.
Finally, ask about failure. Who is alerted when a transform fails, and how long can a delivery stall last before it matters to an investigation or an audit? Also ask what the filtering saves. Cloudflare says dropping discarded records before delivery can cut downstream volume and cost, but it does not publish savings figures.
The takeaway is simple. Cleanup can now happen before delivery rather than after it. A rule that can stop the pipeline needs someone accountable for it.
Produced by the WebPulse Newsroom with AI assistance from the original reporting credited below, and checked against that source by our editorial review. How we use AI.
Original reporting: Cloudflare.
