Microsoft says old bug caused massive Outlook outage
Microsoft said a latent bug, a Teams failover, and reduced capacity caused the outage that hit over 70 countries and lasted nearly 24 hours.
Photo: The VergeKey points
Microsoft said a latent bug, a Teams failover, and reduced capacity caused the outage that hit over 70 countries and lasted nearly 24 hours.
Microsoft revealed that a latent bug, a Teams failover, and reduced server capacity caused the massive outage that took Outlook, Teams, and Copilot offline for nearly 24 hours. The disruption affected more than 70 countries. Charles Lamanna, Microsoft's EVP for Copilot and AI Agents, explained on X that there were three root causes. The company's preliminary post-incident report described the bug as affecting select Windows servers due to lock contention inside an internal Windows process.
Reports of the outage began on November 12, with Exchange Online servers in Switzerland falling offline, according to a source familiar with the incident. The problem spread to Mexico and other countries as failover systems tried to shift email traffic. Within a couple of hours, more than 80 percent of Exchange servers were unresponsive. Microsoft marked the outage as sev0, its highest internal incident level, indicating catastrophic service disruption.
Three root causes
Lamanna said Microsoft typically maintains two network gateway systems that connect separate systems. One of those gateways wasn't working for Microsoft Teams, and it overwhelmed the primary gateway. It also happened that nearly 14 percent of the capacity for that gateway was out of service due to an unrelated issue from the previous day. Those two factors caused that gateway to encounter a latent bug that had been in Microsoft's systems since July.
The bug had been sitting on Microsoft 365 networking gateway servers since July without any impact. The contention issue is worsened on machines with more cores and newer hardware, while Microsoft's load testing focused on weaker hardware with fewer cores, Lamanna wrote. The bug doesn't affect regular versions of Windows since it was the result of a fix prepared specifically for Microsoft 365 networking gateway servers.
Together, the higher load that overwhelmed the gateway, the reduced server capacity, and the latent bug led to a build-up in traffic requests until Microsoft's servers gave out and stopped functioning. This happened on a Thursday afternoon, during peak hours of use in the US. The outage lasted for almost a full day, with services not fully recovered until Saturday morning.
Why recovery was slow
Recovery took longer than usual because as servers returned to service, they were immediately overwhelmed by waiting traffic and became unresponsive again, Lamanna wrote. Microsoft's normally automated recovery processes were not designed to restore service from this broad unresponsive state. Since the main code defect was a latent bug from several months previous, normal rollback, flighting, and feature flags were unable to quickly revert the change.
Microsoft engineers had a running bridge call the entire time, and at one point it hit the maximum of 1,000 participants. The company was even struggling to roll back servers to a healthy state using its usual processes of reverting recent changes. Crash reports and logs quickly ruled out the possibility of an attack, and Microsoft was liaising with DDoS specialists to understand whether this could have been an attack.
The outage affected businesses, hospitals, law enforcement, and other organizations around the world, according to reports. Governments, healthcare systems, schools, courts, airlines, and businesses were affected. Some estimates put lost productivity from the outage near $7 billion, though one analysis suspected the real economic impact would be far larger once downstream business interruption is fully understood.
Economic toll
Lamanna wrote on X, "We are deeply sorry for the impact to all our customers and recognize the impact that this has on their normal business & operations." He also noted that the teams behind Office, Outlook, and Microsoft Teams all started reporting to him last month after the head of Office and Teams, Ryan Roslansky, left the company. Microsoft is now working on a final post-incident review that will be published later this week.
InfoWorld argued that the outage was a cloud architecture failure, not a simple grouping of technologies that had a bad day. The cloud did not magically route around the problem, and automation did not save the business. The article said Microsoft owns this one, but other cloud providers are not immune. It called for architectural changes and operational discipline to reduce dependency on such technology.
Frequently asked questions
What caused the Microsoft outage?
Microsoft said a latent bug, a Teams failover, and reduced server capacity caused the outage. The bug had been on Microsoft 365 networking gateway servers since July, and a Teams failover increased load while 14 percent of capacity was out of service.
How long did the Microsoft outage last?
The outage lasted nearly 24 hours, with services not fully recovered until Saturday morning. It began on November 12 and affected more than 70 countries.
Was the Microsoft outage a cyberattack?
No. Microsoft said it was not a cyberattack and not AI-related. Crash reports and logs ruled out the possibility of an attack, and the company was liaising with DDoS specialists to confirm.


