Skip to content
Business Efficiency

Microsoft's email outage came from a July change that testing missed

A preliminary report says a months-old defect met a failover and lost capacity. Rolling back was not an option.

W
WebPulse Newsroom
AI-assisted · 4 min read
Share on X LinkedIn
Microsoft's email outage came from a July change that testing missed
In brief
  • Microsoft's preliminary report blames last week's Exchange Online outage on a July 2026 change that passed load testing but hid a defect in its network gateway servers.
  • A failover and reduced capacity exposed the flaw, and a rollback was not possible because the change was months old.
  • Ask your own teams whether tests use real hardware, whether recovery covers broad failures, and what the fallback is when email stops.

The cause was months old, so the usual escape route was closed

Outages are often blamed on a recent change. This one was different. Microsoft says the root cause was a change made in July 2026, which sat in its network gateway servers until last week.

The Register reported the findings from Microsoft's preliminary post incident report, published November 15. Microsoft says the update passed load testing and was deployed without any impact. The lesson here is that a change can be tested, shipped and quiet, and still be a hazard. It just has not met the right conditions yet.

How a small defect became a fleet-wide failure

Microsoft describes the defect as "lock contention inside an internal Windows process." In plain words, many tasks queued for the same shared resource. The machine then burned its CPU and memory on waiting instead of working.

Microsoft's own wording is that the contention is "worsened on machines with more cores and newer hardware." Yet its load testing "tended to focus on weaker hardware with fewer cores." By Microsoft's own account, the tests leaned on the wrong kind of machine.

Then two ordinary events coincided. Teams failed over to the primary gateway serving Exchange Online, at 0107 UTC on November 12. Load then built through the day, and Microsoft's monitor first flagged connectivity problems at 1603 UTC. The surge was not unusual in itself.

Meanwhile, almost 14 percent of the gateway's server capacity was out of service while being restored. Microsoft says there was still enough headroom. The report does not say when that capacity was lost.

Neither event would have mattered alone. In combination, they triggered the hidden flaw.

Failover spread the failure instead of containing it

The Register calls this a classic cascading failure. When a server stops responding, traffic moves to the next healthy one. That server hits the same defect, and traffic moves again. Within two hours, the report says, the defect had appeared across the fleet. Microsoft's figure for the worst moment is that over 80 percent of the gateway servers had stopped responding.

Restarting servers did not help. They were overwhelmed again as soon as they came back. Microsoft's report says its "normally automated recovery processes were not designed to restore service from this broad unresponsive state."

Why email stayed down longest

Some services could step away from the faulty gateway. The report names Teams, OneDrive, SharePoint and Office apps. It says others, such as Exchange Online, cannot do this "for scale reasons." Many users experience Exchange Online as Outlook, so for them the outage was email.

A rollback was also off the table, because the defect was months old. Engineers instead built what The Register calls a patch for the patch. In Microsoft's words, the fix put the code path back to "the previously known good state in Windows." Engineers pushed it to servers in batches, so service returned step by step rather than all at once.

The timings show how long this took. Microsoft's monitor flagged connectivity problems at 1603 UTC on November 12, and the first support case arrived at 1633. Microsoft identified connection negotiation as the main culprit by 0030 UTC on November 13. It applied the mitigation at 0720 UTC. It confirmed healthy service at 1610 UTC and sent its final customer update at 0415 UTC on November 14.

What this means for your organisation

This report is preliminary. Microsoft's final report is due November 20, and details may change. Even so, three questions are worth asking now.

First, does your own testing use the hardware you really run? Microsoft's admission about weaker test machines applies to any company that ships software or runs its own infrastructure.

Second, has anyone tested recovery from a broad failure, not just the loss of one server? The report says Microsoft's automated recovery was built for narrower problems.

Third, how long can your business run without email or chat from one vendor? Ask your Microsoft account team what the final report changes, and ask your own IT team what the manual fallback is. The Register's verdict is that Microsoft's release process "deserve[s] a long, hard look." Customers can ask for the same scrutiny of their own.

A rollback button works for recent mistakes. The older the defect, the more the fix looks like surgery. Plan for the failure you did not test.

Produced by the WebPulse Newsroom with AI assistance from the original reporting credited below, and checked against that source by our editorial review. How we use AI.
Original reporting: The Register.