Laravel patched PHP fast where it owned the runtime, slowly where customers did

Patch speed depends on who owns the runtime

A security patch is a race, and you do not choose the start line. Laravel's post mortem of its October response shows how much the start line matters. Laravel credits Cloud's speed to one fact: it controls the runtime, the software layer where customer applications actually execute.

Laravel runs four products: Cloud, Forge, Envoyer and Vapor. The PHP project warned it on the evening of October 20, at 19:58 UTC. Public fixes arrived at 13:38 UTC the next day, less than a day later. They covered two critical remote code execution flaws. Node.js issued ten fixes of its own, one for a flaw in its Brotli decoder, a compression component, that Laravel says could allow remote code execution. Laravel reports no sign that any customer was exploited.

How Laravel patched running applications

Laravel Cloud runs on Kubernetes. Think of Kubernetes as a loop that compares what each workload should look like with what it looks like now, then fixes the difference. Laravel adds its own logic to that loop. One piece, the app operator, manages customer applications.

Laravel changed the operator so it places patched PHP files into each workload as it starts. Customer images were not rebuilt or redeployed. The operator restarts each workload, and Laravel says the platform absorbs the restart with no customer downtime.

Laravel also built its own PHP image, ready at 14:24 UTC. It could not count on the usual public source. The company says the official Docker Hub images only appeared hours after the release. It adds that a few third-party hosts its own builds pull from went offline for part of the day. Building in-house let it start sooner.

A built-in pacer limits how fast the operator restarts workloads, to protect billing and other internal systems. Laravel made it five times faster, which it estimates cut pacing from about three hours to about 35 minutes. It did not remove the pacer entirely. Laravel's concern was that restarting everything at once could swamp those internal systems.

A canary, a small test rollout, began at 11:17 UTC in two low-density regions, before the release was public. The last region reached full coverage at 18:05 UTC. Laravel says Cloud customers had to do nothing. New deployments on Cloud were halted for roughly three and a half hours during the rollout.

Earlier, at 08:55 UTC, Laravel switched on Cloudflare's managed firewall rules. It says a firewall cannot cover every route to a flaw like this. It treats the rules as a way to lower exposure, not a substitute for the fix.

Where customers owned the servers

Forge works differently. Customers must upgrade PHP on each of their own servers. Laravel says the release set off about ten times the normal volume of upgrade jobs. Many failed and retried against unreliable package repositories and outdated third-party sources.

Customer deployments waited in the same line as those jobs, and sat behind them until they timed out. Laravel reports about 48,000 jobs queued at one point. At about 17:45 UTC it told customers the queue was clear. Laravel calls that message premature. Roughly 90 minutes later the backlog grew again, and around 4,800 deployments were cancelled for waiting too long. Those customers had to redeploy themselves.

Envoyer could not be patched in place, so Laravel took it offline at 13:52 UTC. It returned at 22:58 UTC. For Vapor, Laravel first said it would redeploy customer applications. At 19:25 UTC it reversed course and asked customers to do it themselves.

The weaknesses Laravel admits

Laravel's own diagnosis is candid, and it lists several causes. It says the Forge, Envoyer and Vapor teams are smaller and have less monitoring than Cloud. Some of their systems ran software past its support date, which had to be rebuilt instead of patched.

Laravel also says it had no dependable inventory of the software versions running across its applications. That is a separate weakness from the one that made Cloud fast. Laravel does not present it as the reason Cloud was quick. Cloud's fix is to track each application's exact runtime more strictly and remove unsupported versions.

The work also had a human cost. Laravel says a small group carried the response for about 27 hours, handing off across time zones. It adds that owners changed as work moved on, without always keeping the full history. It also lists unclear owners for each communication channel as a cause of muddled messages.

Laravel expects disclosures like these to keep arriving often. Its view is that AI is compressing the gap between a flaw becoming public and attackers trying it. That is Laravel's opinion, not a measurement in this report.

What to ask your team

Start with five questions. Which of our systems can we patch without asking staff or customers to redeploy? Do we hold a current record of the runtime version each application uses, including unsupported ones? If our vendor's patched images arrive late, what is the second route? Who owns each channel for customer messages in an incident? How long do we wait before declaring a recovery?

Laravel is adopting several of these answers. It will keep the in-place patch mechanism and the firewall rules. It will also wait a set period before announcing a recovery, and it will record incident calls. It is writing a policy for workloads it cannot patch or redeploy automatically.

At Laravel, speed came from owning the runtime. The slow parts were queues, ageing software and unclear ownership, and Laravel admits it had no dependable version record.

Produced by the WebPulse Newsroom with AI assistance from the original reporting credited below, and checked against that source by our editorial review. How we use AI.
Original reporting: Laravel.