--- title: "This Week Was a Dress Rehearsal | Blog | Commvault" type: "BlogPosting" language: "en-US" url: "https://www.commvault.com/blogs/this-week-was-a-dress-rehearsal" date: "2026-11-14T18:51:24-05:00" modified: "2026-11-14T18:54:50-05:00" description: "Two failures in 48 hours stopped work and travel across the United States this week. Neither was an attack. Both were self-inflicted." image: "https://www.commvault.com/wp-content/uploads/2026/07/Thumbnail_Blog-Seven-Dashboards-2026.png" authors: - name: "Pranay Ahlawat" jobTitle: "Chief Technology and AI Officer" breadcrumbs: - name: "Commvault Home" url: "https://www.commvault.com/" - name: "Blogs" url: "https://www.commvault.com/blogs" - name: "This week was a dress rehearsal" --- # This Week Was a Dress Rehearsal Two failures in 48 hours stopped work and travel across the United States this week. Neither was an attack. Both were self-inflicted. --- On Thursday, a network change took Microsoft 365 down in more than 70 countries, and full service took about a day and a half to return. One early estimate puts the lost productivity near $7 billion. On Friday, new equipment overloaded aging telecom gear at the FAA, and every departing flight in the country sat on the ground for more than four hours. More than 9,000 flights were cancelled on Friday and another 4,100 on Saturday. --- If routine changes can take out the world’s email and America’s airspace in the same week, think about what someone who means harm could do. Resilience is no longer a backup plan. It is a board-level mandate. --- #### **Big outages are getting bigger, and more common** --- Outages happen all the time. ThousandEyes logged between 200 and 400 network outages a week worldwide in early 2026, and almost nobody noticed. By some measures the ordinary ones are even improving: Uptime Institute has outage rates per site falling five years running. --- The problem is at the top end, where two things are moving at once. --- The big ones are more frequent. Critical outages at the three largest cloud providers rose 52% between 2022 and 2024, and the hours lost rose 51%, to 221 a year. 2025 was quieter. 2026 has not been: three major Microsoft 365 outages and a nationwide ground stop, with six weeks still to go. --- And they are bigger. A bad day used to cost millions. CrowdStrike’s faulty update in July 2024 cost the Fortune 500 an estimated $5.4 billion. This week’s Microsoft outage may cost $7 billion in lost productivity alone. --- Here are five from the last 13 months. None has been traced to an attack. --- ![](/wp-content/uploads/2026/11/Screenshot-2026-11-14-at-5.42.42-PM.png) --- This week’s Microsoft outage was the longest of the five. The FAA’s was the longest nationwide ground stop in 25 years, and only the second since 9/11. --- Here is what they cost, where someone has put a number on it. Nobody has published an estimate for the January and July Microsoft outages. --- | Outage | Estimated economic loss | Insured loss | Estimated by | |---|---|---|---| | Microsoft 365, Nov 2026 | ~$7B in lost productivity (preliminary) | $300–500M | [Anderson Economic Group](https://edition.cnn.com/2026/11/12/business/live-news/microsoft-global-outage-intl-hnk); [Morningstar DBRS](https://www.newsdrum.in/business/microsoft-services-back-after-38-hour-outage-7-billion-productivity-hit-estimated-12809623) | | FAA ground stop, Nov 2026 | Not yet estimated; refunds alone could cost airlines billions | Not yet estimated | [Morningstar DBRS](https://www.theglobeandmail.com/world/article-faa-ground-stop-us-flights-live-updates/) | | AWS us-east-1, Oct 2025 | $500–650M for U.S. companies | $38–581M | [Parametrix](https://www.parametrixinsurance.com/reports-white-papers/aws-us-east-1-outage---what-parametrix-monitored-in-real-time); [CyberCube](https://riskandinsurance.com/aws-outage-loss-estimates-range-from-38m-to-581m-as-cyber-insurers-face-moderate-impact/) | | CrowdStrike update, Jul 2024 | $5.4B for the Fortune 500 | $0.54–1.08B | [Parametrix](https://www.axios.com/2024/07/24/fortune-500-crowdstrike-outage-impact) | --- Insurance covers only a fraction of this. After CrowdStrike, policies covered an estimated 10–20% of Fortune 500 losses. The rest fell on companies that had no say in the failure. Across the Global 2000, Splunk and Oxford Economics put the annual cost of downtime at $400 billion in a 2024 study, about 9% of profits. --- #### **Why this keeps happening** --- Three forces are compounding, and none of them is going away. --- 1. **Concentration.** AWS, Microsoft and Google hold 61% of a cloud market that now runs $168 billion a quarter. Third-party providers account for about two-thirds of publicly reported outages over the past nine years. When one of them has a bad day, thousands of companies have the same bad day at the same hour. 2. **The pace of change.** Both of this week’s failures began with a change someone made on purpose: a network upgrade at Microsoft, new equipment at the FAA. Uptime Institute finds configuration and change management failures are the leading cause of network outages. We are changing complex systems faster than we can understand what the changes will do. 3. **AI.** Cloud spending grew 55% last quarter, and AI now accounts for about a third of it. That is more load on the same shared infrastructure. It is also a new kind of workload: agents that act on production systems on their own authority. Together they add up to what we call AI vulnerability. Gartner expects misconfigured AI in cyber-physical systems to shut down national critical infrastructure in a G20 country by 2028. --- This week’s failures were accidents. People who attack critical infrastructure for a living just watched how far one failure can travel. The defense is the same either way: an architecture that can recover from a bad update can recover from an attack. --- #### **Your data can’t live in only one place** --- Ask your team one question on Monday: if our primary cloud went down tomorrow, where would we restore from, and where would we restore to? For many companies the honest answer is “the same cloud”. Their data, their backups and their recovery path all live with one provider. When that provider fails, there is nothing to do but wait. --- We built Commvault Cloud so that there is always somewhere else to restore from, and somewhere else to restore to: across clouds, from on-premises to cloud, and from cloud back to on-premises. --- | If this goes down | Single-cloud approach | With Commvault | |---|---|---| | One cloud (Azure or AWS) | Wait until the provider is back | Restore from the copy in the other cloud, or from on-premises, and bring workloads up there | | The customer’s data center | Restores depend on whatever is left on site | Restore from the copy in the cloud and run there until the site is back | | Both clouds | No way to recover until one returns | Customers who keep a copy outside the public cloud can restore from it and run recovery in their own data center | --- I’ll be honest about the trade-off. Every extra copy costs money, and every extra environment adds complexity. I [wrote about this at BCG](https://www.bcg.com/publications/2021/navigating-multicloud-strategy): most companies that called themselves multi-cloud were really running different workloads in different clouds, and true portability was rare and expensive. That was the right caution in 2021. The argument has shifted since. --- Cloud-native architecture changed the math. Workloads built on containers and Kubernetes, with infrastructure defined in code and data decoupled from the platform underneath, can be brought up in a second cloud or in a data center without being rebuilt. Not every workload is worth that. But for the handful of services a company cannot operate without, multi-cloud resilience is more achievable today than it has ever been, and after this week the cost of not having it is a lot easier to put a number on. --- This is not a roadmap item for us. Commvault customers today convert virtual machines between hypervisors and clouds as part of a restore, and spin up a clean, isolated recovery environment, on-premises or in the cloud, through automation. Multiple customers run that automation routinely, not just in a crisis. --- Two other things matter as much as having somewhere else to go. --- - **Visibility.** You can’t recover what you can’t see. Too many companies find out where their critical data lives, and what was never protected, in the middle of an outage. Commvault gives customers a continuous view of where their data is, what is protected and what is exposed, before anything breaks. - **AI agents.** Agents now carry business logic, which makes them production applications. They need the same guarantees as any other critical workload, and most organizations aren’t there yet. With AgentRecover, now in early access for Microsoft Foundry and coming to Amazon Bedrock AgentCore in December, we can audit what an agent did and recover both the agent and the applications it changed. --- #### **Three things CISOs and boards should do this week** --- 1. **Find out what fails together.** Take your ten most critical business services and trace each one down to the cloud region and identity provider it depends on. You are looking for shared fate, and the first place to look is your own backups: do they sit in the same cloud as the systems they protect, and can you restore them anywhere else? The board should be able to get a plain answer to one question: which single vendor failure takes us down, and for how long? 2. **Measure your recovery time in a real test.** Decide which services have to be running on day one and what each hour without them costs. Then take a provider away in an unannounced exercise, at least every quarter, and time how long it takes to get back to a clean state. A tabletop does not count. The number to bring to the board is mean time to clean recovery, as measured. 3. **Put AI agents under the same rules as production systems.** Know every agent you run, what data it can reach and what it can change. Require an audit trail and a way to roll back both the agent and whatever it touched. If nobody can tell you what an agent did last Tuesday and how to undo it, it is not ready for more authority. --- #### **Before the next one** --- This week was a dress rehearsal. Nobody was trying to hurt us, and it still cost billions. Next time someone may be trying, or an AI agent may make the change. Design for that now. That means continuous recovery: in place before the next incident, not assembled during it. --- --- --- **Sources** --- - [2026 Microsoft outage — Wikipedia](https://en.wikipedia.org/wiki/2026_Microsoft_outage) - [What caused the Microsoft 365 outage of November 12, 2026 — Control Plane](https://controlplane.com/blog/post/microsoft-365-outage-november-2026-cause) - [Officials blame aging FAA telecom gear — GeekWire](https://www.geekwire.com/2026/dhs-chief-sees-no-microsoft-link-to-faa-ground-stop-says-possible-ai-role-being-investigated/) - [FAA lifts nationwide ground stop — CBS News](https://www.cbsnews.com/news/faa-issues-nationwide-ground-stop/) - [Travelers grounded in DFW scramble — CBS Texas](https://www.cbsnews.com/texas/news/travelers-dfw-scramble-nationwide-ground-stop-111326/) - [FAA ground stop and Microsoft outage: did AI play a role? — KingyAI](https://kingy.ai/blog/faa-ground-stop-microsoft-outage-ai-connection) - [Nationwide disruptions expose risks of technology dependence — The National Desk](https://thenationaldesk.com/news/americas-news-now/nationwide-disruptions-expose-risks-of-technology-dependence) - [Annual Outage Analysis 2026 — Uptime Institute](https://intelligence.uptimeinstitute.com/resource/annual-outage-analysis-2026) - [Cloud market Q3 2026 — Synergy Research via TechInsights](https://www.techinsights.com/node/61958) - [AWS us-east-1 outage postmortem — BleepingComputer](https://www.bleepingcomputer.com/news/technology/amazon-this-weeks-aws-outage-was-caused-by-major-dns-failure/) - [Microsoft 365 outage drags on for nearly 10 hours — The Register](https://www.theregister.com/2026/01/23/microsoft_365_outage/) - [Microsoft 365 services outage, July 23, 2026 — Cybersecurity News](https://cybersecuritynews.com/microsoft-365-services-outage/) - [CrowdStrike outage cost the Fortune 500 $5.4B — Axios / Parametrix](https://www.axios.com/2024/07/24/fortune-500-crowdstrike-outage-impact) - [The Hidden Costs of Downtime — Splunk and Oxford Economics](https://www.splunk.com/en_us/perspectives/uncovering-downtime.html) - [Gartner: misconfigured AI will shut down critical infrastructure in a G20 country by 2028](https://www.cxodigitalpulse.com/gartner-predicts-that-by-2028-misconfigured-ai-will-shut-down-national-critical-infrastructure-in-a-g20-country/) - [Microsoft outage could cost $7B in lost productivity — CNN](https://edition.cnn.com/2026/11/12/business/live-news/microsoft-global-outage-intl-hnk) - [Microsoft services back after 38-hour outage; insured losses $300–500M — NewsDrum / Reuters](https://www.newsdrum.in/business/microsoft-services-back-after-38-hour-outage-7-billion-productivity-hit-estimated-12809623) - [Microsoft 365 restored after global outage — Bright Defense](https://www.brightdefense.com/news/microsoft-outage-sparks-global-chaos/) - [FAA ground stop live updates; Morningstar DBRS on refunds — The Globe and Mail](https://www.theglobeandmail.com/world/article-faa-ground-stop-us-flights-live-updates/) - [FAA ground stop lasted 4 hours 21 minutes — Informed Clearly](https://informedclearly.com/en/accident/65655/faa-ground-stop-flights-resume-2026) - [AWS us-east-1 outage: U.S. losses of $500–650M — Parametrix](https://www.parametrixinsurance.com/reports-white-papers/aws-us-east-1-outage---what-parametrix-monitored-in-real-time) - [AWS outage insured loss estimate $38–581M — Risk & Insurance / CyberCube](https://riskandinsurance.com/aws-outage-loss-estimates-range-from-38m-to-581m-as-cyber-insurers-face-moderate-impact/) - [Critical cloud outages up 52% since 2022 — CXOToday / Parametrix](https://cxotoday.com/news-analysis/have-cloud-outages-become-more-frequent-or-is-the-internet-more-dependent/) - [Critical cloud outage events down in 2025 — Parametrix](https://www.parametrixinsurance.com/in-the-news/critical-cloud-outage-events-down-in-2025-but-outage-risk-remains-significant) - [ThousandEyes Q1 2026 weekly outage counts — DEV](https://dev.to/firstpasslab/386-global-outages-in-one-week-what-thousandeyes-q1-2026-data-reveals-about-modern-network-526c)