Skip to main content

Kaer NewsSoftware

Microsoft says July bug caused global 365 outage, 80% servers down

Microsoft blamed a July software bug for the November 12 outage that left more than 80% of its networking gateway servers unresponsive.

4 min read
Microsoft says July bug caused global 365 outage, 80% servers downPhoto: BleepingComputer

Key points

A July software bug, triggered by a Teams failover and reduced capacity, caused the November 12 global Microsoft 365 outage, leaving 80% of servers unresponsive.

Microsoft has blamed a software bug introduced in July for last week's global Microsoft 365 outage, which left more than 80% of its networking gateway servers unresponsive. The nearly 24-hour outage on November 12 disrupted services worldwide, including Outlook, Exchange Online, Teams, and Copilot, leaving users unable to access their email and causing significant delays in message delivery.

In a preliminary post-incident report shared in the Microsoft 365 Admin Center, the company said the outage was caused by three problems: a previously undetected software bug, increased traffic after a Microsoft Teams failover, and reduced server capacity. The incident began at 11:20 AM ET on Thursday, November 12, and continued until 11:10 AM ET the following day, when Microsoft confirmed that services were fully restored.

What caused the failure

According to Microsoft, the software bug was introduced in July through an update developed for Microsoft 365 networking gateway servers. While the update had been deployed for more than three months without issues, it contained a bug that could overwhelm servers under certain traffic conditions, causing them to stop responding. Microsoft says its tests of the update did not detect the bug as it used less powerful testing hardware than the production servers.

"Our normal load testing did not detect the latent issue because the traffic mix and SKUs used in our labs," Microsoft explained in its preliminary report. "The contention issue is worsened on machines with more cores and newer hardware; whereas our load testing tended to focus on weaker hardware with fewer cores." The defect caused "lock contention inside an internal Windows process which caused the machine's resources (CPU & memory) to be exhausted."

The bug was eventually triggered after Microsoft Teams automatically switched to the networking infrastructure used by Exchange Online approximately 15 hours before the outage, increasing traffic on those servers. Microsoft Teams also experienced a separate outage earlier that day, but it is unclear whether the two incidents are related. To make matters worse, Microsoft says nearly 14% of the gateway's server capacity was already offline due to an unrelated issue.

Why testing missed it

While the infrastructure had sufficient capacity to handle the reduction, the combination of increased traffic and reduced capacity triggered the software bug. As servers stopped responding, Microsoft's systems redirected users to other available servers, which also became overwhelmed by the additional traffic and failed as well. "In less than two hours, the previously unknown defect was encountered across our fleet and rendered them unresponsive. At its lowest, more than 80% of servers were unresponsive," explained Microsoft.

Microsoft says safeguards designed to prevent problems from spreading between regions also failed to contain the cascading outages, allowing the disruption to spread worldwide. While other Microsoft 365 services, like Teams, OneDrive, and SharePoint, could switch to alternative infrastructure and recover earlier, Exchange Online could not make the same switch, leaving users unable to access their mailboxes. Microsoft says recovery was difficult because restarting affected servers caused them to be quickly overwhelmed again by backed-up traffic.

After identifying the bug, Microsoft engineers developed a fix and deployed it across affected servers, allowing services to stabilize throughout Friday. Microsoft confirmed that active customer impact had ended by 11:10 AM ET on November 13. The company says it has since increased capacity, improved monitoring, and added safeguards to prevent traffic redirection from triggering similar cascading failures.

How recovery unfolded

In a separate statement on X, Charles Lamanna, Microsoft's executive vice president of Copilot, Agents, and Platform, apologized for the disruption. "We are deeply sorry for the impact to all our customers and recognize the impact that this has on their normal business & operations," Lamanna said. Microsoft says its investigation is continuing, with a final post-incident report expected on November 20.

The outage also had real-world consequences beyond email access. At Hotel Ling Bao in Brühl, Germany, a malfunctioning automatic dosing system pumped too much chlorine into the swimming pool, and an email warning about dangerous levels never reached staff because of the outage. Paramedics checked over 15 people at the hotel, and five were taken to hospital; four suffered minor injuries, while one was moderately injured and needed further treatment.

Frequently asked questions

What caused the Microsoft 365 outage on November 12?

Microsoft says a software bug introduced in July, combined with increased traffic from a Teams failover and reduced server capacity, triggered a cascading failure that left more than 80% of networking gateway servers unresponsive.

When was the Microsoft 365 outage resolved?

Microsoft confirmed active customer impact ended at 11:10 AM ET on November 13. The company deployed a fix that reverted the defective change, and services stabilized throughout Friday.

Why didn't Microsoft's testing catch the bug?

Microsoft's load testing used weaker hardware with fewer cores, while the bug was worsened on machines with more cores and newer hardware. The company said the traffic mix and SKUs in its labs did not detect the latent issue.

More from this edition