Late into the night Thursday, hundreds of Microsoft engineers and top executives were sitting in their darkened offices or home basements, logged into a video conference triaging a nightmare scenario for a technology company.
The world still runs on email, and much of that is processed through Microsoft Outlook. For hours, a company outage affected hospitals, schools, businesses and Congressional offices in Washington. Even Microsoft’s own staff couldn’t use email.
On the videoconference, Scott Guthrie, the senior most executive overseeing the company’s computing infrastructure, was proposing possible fixes, according to a person familiar with the call who was not authorized to speak publicly. People chewed on fingernails, rubbed their foreheads and debated how to turn off other services to prevent a “stampeding herd” so that as systems come back online, they would not be overwhelmed with new demands and shut back down.
By Friday morning, the company had apologized at least twice. It said it determined that the problem stemmed from a recent upgrade to its network, which shut down some servers, and that the systems were steadily improving. “Given the scale of demand, we restored services gradually, validating stability at each stage,” the company said in a statement. “While this took longer, it helped ensure a stable restoration.”
Finally, they had kept the herd at bay.
The prolonged and widespread outage was just the most recent reminder that the technological underpinnings of modern life and business are run through just a handful of major companies. And while cyberattacks and malicious actors can target their systems, sometimes the companies learn the hard way that they can introduce devastating flaws when updating or maintaining their own global systems.
Hospitals, law enforcement and government agencies and other vital services are often left vulnerable — unable to function for hours or days at a time.
Last year, Amazon’s cloud computing services were down for 15 hours and knocked a slew of apps and websites, including Zoom and parts of The New York Times, offline. And in the summer of 2024, a faulty software update by the cybersecurity firm CrowdStrike caused airlines to ground flights, businesses to close early, and surgeries to be rescheduled.
For Microsoft, the problems from this week’s outage are not done yet. The short-term financial fallout should be “manageable,” analysts at the investment bank William Blair wrote in a research note Friday. But they said this prolonged outage is a reputational hit that could make companies and other customers reconsider having so much riding on one technology provider.
“We believe the outage could prompt enterprises to reassess their dependence on Microsoft’s infrastructure,” they wrote.
Outlook and other services affected, including Teams and its A.I. assistant Copilot, are part of Microsoft 365’s cloud subscription business, which is used by more than 450 million people and brought in almost $28 billion in its last quarter, which ended Sept. 30. Online memes joked the offering should be rebranded as Microsoft 364.
These subscriptions are central to the company’s business and the foundation on which it is promoting A.I. tools. Microsoft makes the case to customers that if they keep more data on Microsoft’s systems, they will be able to implement and maximize the benefits of A.I. more easily. Microsoft acknowledged that about 20 different services were impaired by the outage, a reminder that consolidating vendors can also create a bigger single point of failure.
Microsoft first detected the Outlook problems Thursday morning, around 9 a.m. on the West Coast, where the company is headquartered. It soon spread from the Americas around the world.
At first, the company said it thought the cause was an unexpected spike in traffic. Then it determined a large number of the company’s servers in its data centers were down, straining the rest of the system. Even as some servers were brought back online, they would shut down again, like a game of Whac-A-Mole.
Luca Arnaboldi, an assistant professor of cybersecurity at the University of Birmingham in England, said the outage was a reminder that when people use “the cloud,” they are actually relying on servers, which have finite resources.
On the overnight videoconference, Mr. Guthrie said while other systems were down too, the teams should prioritize restoring Outlook. Charles Lamanna, who heads many of Microsoft’s productivity services, was there too. He had already publicly apologized and said Microsoft had ruled out a cyberattack as the culprit.
The company’s own systems were to blame.
Isabella Kwai contributed reporting from London. Michael Levenson contributed reporting from New York.
