- Google Project Zero says testing and delivery, not triage or coding, usually limit how fast vendors ship an emergency patch.
- It describes three fast lanes used by large vendors: feature flags, input filtering and alternate update channels. Each has trade-offs.
- Ask vendors how they would fix an urgent flaw, and how they verify emergency updates.
When a flaw is being exploited, how fast the fix can arrive is largely set by how the vendor built its update system. That is our reading of a new post from Google Project Zero, and it changes what you should ask your suppliers.
The slow part is not the code
Project Zero says some vendors worry that they could not fix a flaw that is causing immediate harm to users. The cause is limits in their patch delivery systems. The team wrote the post to share what it has seen across vendor discussions and security reviews.
It describes a typical patch as six stages: triage, patch development, testing, partner review, delivery and activation. Activation can mean a restart.
In an emergency, the researchers say, triage is usually fast and developers can move a fix up the queue. Partner agreements often allow exceptions too. Delays in those stages are rare. They happened when a flaw was unusually complex, or when a security team lacked a full picture of its software components and who maintains them.
Testing and delivery are where most vendors lose time. Project Zero says these stages matter more when the timeline is short.
Why careful testing slows everything
Testing is slow for a reason. A bad update can 'brick' a device, leaving it unable to do its main job or to receive a repair update. Updates have also corrupted or lost user data. Any loss of function after a security update makes users less willing to install the next one.
The cost depends on the product. A broken mobile app can be replaced through the app store, and the data usually sits on a server. A bricked phone must go back to the maker or the shop. Project Zero says vendors often face a trade-off between patching quickly and avoiding buggy patches.
Delivery adds friction. Devices that check for updates at set intervals cannot be patched faster than that interval. Users delay patches that need their action, and restarts are unpopular. Project Zero notes that Chrome and Microsoft have written about this problem.
Three fast lanes large vendors use
The first is the feature flag. This is a switch in the software that a remote server can flip. It lets a vendor turn a feature off without shipping new code. Project Zero cites Apple's 2019 FaceTime flaw, where Group FaceTime was switched off this way. The key benefit is that testing happens in advance, with each flag position tried before any emergency.
Meta applied the idea to a whole library. It compiled two versions of the WebRTC video-calling library into one program. A flag chooses which one runs. If a new version misbehaves, users move back to the old one. Project Zero suggests the same pattern could pair two different libraries that do the same job, so one can be switched off when it has a flaw.
The second lane is filtering. A vendor sends a rule that blocks the specific input needed to trigger a flaw. Android's Intent Firewall works this way, and it was recently used against flaws in third-party Android wallets. Security tools such as Microsoft Defender and Google Play Protect can also carry such rules. Filtering covers cases nobody planned for, but it can slow software down. New filters also need testing that cannot be done before the flaw is known.
The third lane is a separate update channel. Android's APEX updates high-risk components without a full system update. Some apps download single libraries outside the normal cycle. Linux Livepatch and Windows hotpatch replace functions in running code without a restart.
Some fast lanes are an attack surface too
The risks differ by lane. Feature flags are limited by coverage, not exposure. A vendor must decide in advance which features can be switched off. If that list is incomplete, a flaw may fall outside it.
Other lanes carry security risk. Project Zero warns that apps which download libraries must check that the files really come from the vendor. Otherwise the channel can introduce vulnerabilities of its own. Hotpatching needs memory that can be written to and run at the same time, which can open ways around exploit protections. It also does nothing for the testing problem.
Security tools that ship blocking rules also have to read all kinds of untrusted files and data, frequently with elevated system rights. Project Zero cautions that this makes them a risk in their own right, even though they can serve as an emergency route where they are already installed.
The lesson here is that a rescue route is software too. It needs the same security review as the product it protects.
Why Project Zero urges planning ahead
The researchers state that large language models (LLMs) are increasing the discovery and exploitation skills of both attackers and defenders. More actors can now run novel attacks at greater speed. Project Zero argues that emergency tools need not be heavy or fix every bug. They need to handle the likeliest and most severe flaws while keeping devices reasonably usable.
This post is guidance drawn from large vendors, not a count of how many firms have these systems. It does not say how many lack them.
What to ask your vendors and your own team
Ask each critical supplier a plain question: if a flaw were being exploited today, what is the fastest safe way you could protect us? The answer should name a mechanism, not a promise.
Ask whether risky features, such as media decoding on untrusted input, can be turned off remotely. Ask whether they can block specific input by rule. Ask how emergency downloads are verified as genuine.
For software you build yourself, ask whether security staff know every component and who owns it. Project Zero has seen that gap delay urgent fixes in rare cases.
In our reading, how fast a vendor can fix an urgent flaw is largely a design choice, made before the flaw arrives. Project Zero notes that triage and development can matter in some cases, but testing and delivery limit most vendors. It is easier to learn that answer in a planning meeting than in the middle of an incident.
Produced by the WebPulse Newsroom with AI assistance from the original reporting credited below, and checked against that source by our editorial review. How we use AI.
Original reporting: Google Project Zero.
