[AWS Outage 2025] Part 1: The Diwali That Broke the Cloud — Inside AWS’s DNS Race Condition Disaster 🎆
This Diwali, it wasn’t just firecrackers that burst — the cloud did too. 💥
On October 20, 2025, while millions in India were celebrating the festival of lights, a tiny timing bug in Amazon Web Services’ infrastructure decided to throw its own fireworks display. Except these fireworks took down one of the most important pieces of internet infrastructure on the planet, affecting thousands of companies and millions of users worldwide. No deployment gone wrong. No misconfiguration. No hacker attack. Just a race condition — a bug so sneaky that it had been hiding in production code for who knows how long, patiently waiting for the perfect storm of timing to trigger it. ⏰
This is the story of how that storm arrived, and why it brought DynamoDB — the database service that powers a huge chunk of AWS — to its knees.
🏗️ Why DynamoDB Matters (And Why You Should Care)
Before we dive into what went wrong, let’s understand what DynamoDB actually is and why its failure was such a big deal.
Think of DynamoDB as the foundation of a massive skyscraper. It’s Amazon’s NoSQL database service — a place where applications store data that they need to access quickly, reliably, and at massive scale. This service handles over 1 billion database requests per hour globally. That’s roughly 277,000 requests every second. 🚀
But here’s the really important part: DynamoDB isn’t just used by customers building their apps. Hundreds of other AWS services depend on DynamoDB to function. Services like Lambda (for running code), EC2 (for virtual servers), IAM (for logging in), and dozens of others all rely on DynamoDB working correctly. So when DynamoDB goes down, it’s like pulling the foundation out from under that skyscraper — everything starts collapsing. 🏢💔
In the US-EAST-1 region alone (that’s AWS’s massive data center in Northern Virginia), DynamoDB manages hundreds of thousands of DNS records. To handle this enormous scale without human operators manually updating things, AWS built a sophisticated automated system to manage all these DNS records 24/7.
And that automated system is where our story really begins.
🧠 The Architecture: Two Independent Systems That Should Never Fail
AWS designed DynamoDB’s DNS management with two separate components, specifically so that if one failed, the other could keep things running. Let me explain these in simple terms.
The DNS Planner — The Brain 🎯
The DNS Planner is like a supervisor constantly monitoring traffic. It watches the health of hundreds of load balancers (servers that distribute traffic) and decides which ones should receive customer requests.
Every few minutes, it creates a new “plan” — basically a list that says “send traffic to these servers with these percentages.” Each plan gets a version number (like version 100, 101, 102, etc.). The Planner keeps creating new plans as conditions change: servers go down, new capacity comes online, traffic patterns shift.
The DNS Enactor — The Worker 👷
The DNS Enactor is like a worker that takes the Planner’s instructions and actually implements them. It picks up the latest plan and updates Route 53 (Amazon’s DNS service) with the new list of servers.
To make sure things stay running even if one worker fails, AWS runs three completely independent copies of the DNS Enactor, each in a different data center (called an Availability Zone). Each one operates independently — they don’t talk to each other, they just each try to apply the latest plan.
AWS designed these Enactors to use something called “transactions” when updating DNS. This is supposed to prevent problems when two Enactors try to update the same thing at the same time. The transaction system ensures that only one can write at a time, preventing chaos. ✅
Here’s where things get interesting: before an Enactor starts its work, it checks “Is my plan newer than what’s currently applied?” This check is supposed to prevent old plans from overwriting new ones.
Spoiler alert: that check had a fatal flaw. 🚨
⏱️ The Race Condition: When Timing Becomes Everything
Now we get to the heart of the problem. This is where things went catastrophically wrong due to a series of events happening in a very specific order, with very specific timing.
The Setup: Everything Looks Normal ✨
It’s October 19, 2025, at around 11:48 PM Pacific Time (that’s October 20 at 10:18 AM in India). The DNS system is humming along normally. Plans are being created, applied, and everything is working fine.
Then something unusual happens.
Act 1: Enactor A Gets Stuck in Traffic 🚗💨
One of the three DNS Enactors (let’s call it Enactor A) picks up an older plan — let’s say it’s version 100. Enactor A performs its check: “Is version 100 newer than what’s currently applied?” The answer is yes, so it starts working.
But here’s where things get weird: Enactor A starts experiencing unusually high delays as it tries to update DNS endpoints. It’s like a delivery driver getting stuck in terrible traffic. The Enactor keeps retrying, but it’s slow. Really slow. 🐌
Meanwhile, life goes on for everyone else.
Act 2: The Planner Keeps Working, Oblivious 📝
While Enactor A is stuck in its traffic jam, the DNS Planner is still doing its job. It doesn’t know or care that Enactor A is delayed. It keeps monitoring servers, keeps creating new plans. Version 101. Version 102. Version 103. Version 104. Version 105.
Each new plan reflects the current state of the world, getting fresher and fresher.
Act 3: Enactor B Speeds Through 🏎️
Another DNS Enactor (Enactor B), running in a completely different data center, picks up one of these newer plans — let’s say version 105.
Enactor B isn’t experiencing any delays. It zips through all the endpoints quickly, updating each one with version 105. Everything looks great. Within minutes, all DNS records are updated with the newest, freshest plan.
Job done, right? ✅
Act 4: Cleanup Time (The Setup for Disaster) 🧹
After Enactor B finishes its work, it does what it’s programmed to do: it triggers a cleanup process. This cleanup looks at all the old plans sitting around and says “These are way too old compared to version 105. Let me delete them to keep things tidy.”
The cleanup process identifies plan version 100 (the one Enactor A is still slowly working on) as ancient history. It marks version 100 for deletion.
But here’s the critical moment: At the exact same time, Enactor A finally finishes fighting through its delays and successfully applies version 100 to the main DynamoDB endpoint (dynamodb.us-east-1.amazonaws.com). ⚠️
Remember that version check Enactor A did at the start of its work? That check said “version 100 is newer than what’s applied.” But that check was performed hours ago in computer time. By now, version 105 is actually what’s applied. But Enactor A doesn’t know that — it never checked again. It just trusted its old check and wrote version 100, overwriting the newer version 105.
Act 5: The Fatal Blow 💀
Now the cleanup process executes. It sees version 100 as the active plan, but it was already marked for deletion because it’s so old compared to version 105.
The cleanup deletes version 100.
Result: The DNS record for dynamodb.us-east-1.amazonaws.com becomes completely empty—pointing to zero IP addresses. No servers. Nowhere to send traffic. DynamoDB effectively vanishes from the internet. 🫥
Even worse, the deletion leaves the entire DNS management system in a confused, inconsistent state. The automated recovery processes can’t fix it because they don’t have a valid plan to apply. The system is locked up, requiring human operators to manually intervene and fix things.
This is a textbook race condition: everything depends on precise timing. If Enactor A had been a few seconds faster or slower, nothing would have happened. If Enactor B had picked up a different plan, nothing would have happened. But with this exact sequence, at this exact timing, disaster struck. 💥
🔥 The Immediate Impact: DNS Resolution Fails
At exactly 11:48 PM PDT on October 19 (10:18 AM IST on October 20), the moment that empty DNS record went live, the cascade began.
Every system trying to make a new connection to DynamoDB immediately hit a wall. When they asked “What’s the IP address for dynamodb.us-east-1.amazonaws.com?” the DNS system responded with… nothing. An empty answer. It’s like calling information to get a phone number and the operator saying “That number doesn’t exist.” 📞❌
Here’s what happened:
Existing connections kept working ✅ — If your application already had an active connection to DynamoDB before 11:48 PM, you were fine. Those connections kept humming along normally.
New connections failed instantly ❌ — Any application trying to open a new connection (or reconnect after a timeout) found DynamoDB completely unreachable.
This affected:
Customer applications trying to read/write to their DynamoDB tables 📊
Internal AWS services that depend on DynamoDB ⚙️
Any system that had to reconnect for any reason 🔌
Within minutes, AWS’s internal monitoring systems started lighting up like a Christmas tree. Alarms everywhere. Services failing across the board. 🚨🚨🚨
🛠️ The Recovery Timeline: Every Minute Counted
Let’s walk through what happened next, minute by minute (all times in Pacific Time with Indian Standard Time in parentheses):
11:48 PM PDT Oct 19 (10:18 AM IST Oct 20) ⏰: The race condition triggers. DynamoDB’s DNS record becomes empty. All new connections to DynamoDB start failing immediately.
12:38 AM PDT Oct 20 (1:08 PM IST) 🔍: AWS’s incident response teams have been scrambling for 50 minutes. They finally identify the root cause — DynamoDB’s DNS state is corrupted. Now they know what they’re dealing with.
1:15 AM PDT (1:45 PM IST) 🔧: Engineers apply temporary fixes that allow some internal AWS tools to connect to DynamoDB. This is critical because they need these tools to diagnose and fix other problems. It’s like making sure the firefighters have water before they can fight the fire.
2:25 AM PDT (3:55 PM IST) 🎉: Success! Operators manually correct the DNS records in Route 53. All DNS information is now accurate again. However, customers worldwide still have the old (empty) DNS information cached on their computers and servers.
2:25 AM — 2:40 AM PDT (3:55 PM — 4:10 PM IST) ⏳: This is the DNS propagation window. As cached DNS entries expire across the world (typically DNS records have a “time to live” of a few minutes), systems perform fresh DNS lookups and get the correct IP addresses. New connections to DynamoDB start succeeding.
2:32 AM PDT (4:02 PM IST) 🌍: DynamoDB’s global table replicas (copies in other regions) have fully caught up with all the delayed updates.
2:40 AM PDT (4:10 PM IST) ✅: DynamoDB’s primary disruption officially ends. The database is accessible again. Approximately 2 hours and 52 minutes have passed since the initial failure.
Customers can breathe a sigh of relief, right?
Not quite. 😬
💣 The Hidden Time Bomb: EC2 Was Already Broken
Here’s the thing about complex systems: failures cascade in ways you don’t expect. While DynamoDB was down, other systems that depended on it had started breaking in their own unique ways.
The biggest problem? EC2’s DropletWorkflow Manager (DWFM) system had been failing health checks for nearly 3 hours because those checks required DynamoDB. When DynamoDB came back online, DWFM tried to re-establish connections to tens of thousands of servers simultaneously. 🌊
This created a congestive collapse — imagine every car in a city trying to drive through a single intersection at the same time. DWFM couldn’t make progress because it was drowning in its own work. And this problem would last many more hours, affecting EC2 instance launches, which would then affect dozens of other services.
But that story — the cascade through EC2, Network Manager, Load Balancers, Lambda, and dozens of other services — is for Part 2 of this series. 📖
🎓 The Technical Lesson: Why This Race Condition Slipped Through
For the engineers reading this, here’s what went wrong from a software design perspective:
1. The Version Check Happened Too Early ⏰
The DNS Enactor checked “Is my plan newer?” at the beginning of its work. But if that work takes hours due to delays, the check becomes meaningless. Modern distributed systems typically use “compare-and-swap” operations that verify the version immediately before writing, not hours earlier.
Think of it like this: you check your bank balance, see you have $100, go shopping for 3 hours, then write a check for $90 assuming you still have $100. But during those 3 hours, your rent auto-payment already took $95. Your check bounces. You needed to check your balance again right before writing the check. 💳
2. No Protection Against Stale Writes 🛡️
The Route 53 transaction system prevented two Enactors from writing simultaneously (which is good), but it didn’t prevent one Enactor from writing stale data (which is bad). The system needed version checking at the database level, not just at the application level.
3. No Recovery Procedure for Inconsistent State 🚑
When the cleanup process deleted the active plan, it left the system in a broken state that the automation couldn’t fix. There was no “Oh no, something’s wrong, let me reset to a safe state” procedure. Engineers had to manually intervene because no automated recovery path existed for this specific failure mode.
4. Testing Didn’t Cover High-Delay Scenarios 🧪
AWS’s test suite didn’t include scenarios where one Enactor experiences abnormally high delays while others operate normally. Testing for timing-dependent bugs is notoriously difficult, but it’s essential for distributed systems where timing is everything.
🔮 What’s Coming Next
This DNS failure was just the first domino. Over the next 30 hours, a cascade of failures rippled through AWS’s infrastructure:
EC2’s congestive collapse (couldn’t launch new instances for hours) 🖥️
Network Manager’s backlog (instances launching without network connectivity) 🌐
Load Balancer health check failures (causing perfectly healthy services to be marked as failed) ⚖️
Lambda, ECS, EKS, Fargate, Connect, STS, IAM, and Redshift all experiencing their own unique failure modes ⚠️
In Part 2, we’ll dive into the EC2 congestive collapse — how DynamoDB’s recovery paradoxically made things worse for EC2, and why it took engineers nearly 12 hours to manually rescue the system from a death spiral of its own making. 🌀
📊 Verified Timeline Summary (From Official AWS Report)
Time (PDT) | Time (IST) | Event
------------------+--------------------+--------------------------------
Oct 19, 11:48 PM | Oct 20, 10:18 AM | 💥 DNS race condition
| | triggers; DNS record becomes
| | empty
Oct 20, 12:38 AM | Oct 20, 1:08 PM | 🔍 AWS identifies DynamoDB DNS
| | corruption as root cause
Oct 20, 1:15 AM | Oct 20, 1:45 PM | 🔧 Temporary fixes applied;
| | internal tools reconnected
Oct 20, 2:25 AM | Oct 20, 3:55 PM | 🎉 All DNS records manually
| | restored by operators
Oct 20, 2:32 AM | Oct 20, 4:02 PM | 🌍 Global table replicas fully
| | caught up
Oct 20, 2:40 AM | Oct 20, 4:10 PM | ✅ DynamoDB primary disruption
| | ends Next in Part 2: The congestive collapse — when recovery triggers catastrophe. 🌊
📚 References
All technical details and timeline information in this blog are sourced from the official AWS Post-Event Summary: