- Cloudflare says a Hyperdrive MySQL caching flaw caused empty or hung query responses about 0.001% of the time during the beta, only with caching on.
- The cause was a Postgres-era assumption: the Client cached the bytes it sent the Worker, which differed from the format the Endpoint expected.
- Ask teams what their caches store, which assumptions were carried across databases, and whether monitoring catches quiet wrong answers.
A wrong answer that looks normal
An error is easy to see. A database that returns an empty result when data exists is harder to catch. The application receives a valid reply and carries on.
Cloudflare described one such failure on October 14, as Hyperdrive support for MySQL reached general availability. Hyperdrive is Cloudflare's database connection pooler for Workers. It keeps warm database connections open and shares them across requests. It can also cache common query results.
During the beta, Cloudflare's highest-volume MySQL customer reported a small but steady rate of empty responses where data was expected. The lesson is about assumptions. A shortcut that is safe in one system can quietly fail in the next.
How a cache served the wrong format
Every Hyperdrive instance has two parts. The Client node runs beside the customer's Worker. The Endpoint node holds the connection pool and sits near the origin database. Each node uses the Cloudflare cache in its own data center. If both run in the same data center, they share one cache.
Sharing helps speed. A Client can answer a query straight from results the Endpoint just stored. This only works if the same bytes are cached for the same answer, whichever node stored them.
MySQL clients and servers agree on optional features, called capabilities, when a connection opens. One of them, CLIENT_DEPRECATE_EOF, changes how the end of a result set is marked. Cloudflare found that mysql2, a popular Node.js MySQL library, does not support it.
Hyperdrive supports it. So when a Worker's driver does not, Hyperdrive rewrites the origin's reply into the older format that driver expects.
Here was the flaw. The Client's caching code was built for Postgres. It assumed it could cache exactly the bytes it sent back to the Worker. With MySQL, those bytes can be in the Worker's format, while the Endpoint expects Hyperdrive's own format.
Picture a shared filing cabinet. One clerk files documents in English. Another reads them expecting French. Most days the clerks never reach for the same folder. Occasionally they do.
Cloudflare described two failure paths. In one, the Endpoint caches a reply in Hyperdrive's format, and a Client sends it to the Worker without translating it. In the other, a Client caches a reply already translated for the Worker, and an Endpoint hands it to a different Client. That Client tries to translate it again. The result was an early end-of-data marker or a hang.
The bug needed a specific setup: caching on, Client and Endpoint in the same data center, and one reading what the other had cached. That explains why it was so rare.
Why it took days to find
Cloudflare tracks how often client connections end in a Hyperdrive-related error. This bug did not show up as a clean error. Engineers combed telemetry, raised the trace rate for that customer's instance and searched for network faults. Nothing explained it.
The clue was that the bug appeared only where caching was enabled. Cloudflare then studied the MySQL protocol for assumptions carried over from Postgres. After days of work, the Client caching assumption came to light.
The fix was to store one canonical version of each result in the cache, encoded with Hyperdrive's own capabilities, whichever node writes it. Cloudflare says the empty-row problem then went away. Its Client connection success rate reached 99.99%.
What leaders should ask their teams
This is one vendor's account of one bug. The post does not say how many customers were affected. Still, the pattern applies to any stack with a pooler, proxy or cache between an application and its data.
First, ask what the cache stores. Is it what the system received, or what it sent to a particular caller? Those can differ. Caches shared across components need one agreed format.
Second, ask which assumptions were carried over from another system. Cloudflare's flaw came from Postgres habits applied to MySQL. A migration or a new database engine is the moment to retest them.
Third, ask whether monitoring covers quiet failures. Error rates did not catch this. Checks on unexpected empty results, or on results that differ between cached and uncached paths, might.
Fourth, ask how customer reports reach engineers. Cloudflare says customer feedback during the beta led to the discovery. Treat a small, steady complaint from a heavy user as data.
That cost is why poolers and caches exist. Each speed-up adds a place where two components must agree on exactly what the data looks like. When they disagree, the system can still answer, and the answer is wrong.
Produced by the WebPulse Newsroom with AI assistance from the original reporting credited below, and checked against that source by our editorial review. How we use AI.
Original reporting: Cloudflare.
