But: given finite resources, should you respond to this incident by auditing your DNS management systems (or all your systems) for race conditions? Or should you instead figure out how to make the Droplet Manager survive (in some degraded state) a partition from DynamoDB without entering congestive collapse? Is the right response an identification of the "most faulty components" and a project plan to improve them? Or is it closing the human expertise/process gap that prevented them from throttling DWFM for 4.5 hours?
Cook isn't telling you how to solve problems; he's asking you to change how you think about problems, so you don't rathole in obvious local extrema instead of being guided by the bigger picture.
Even you can't help it - "enumerating a list of questions" is a very engineering thing to do.
Normal people don't talk or think like that. The way Cook is asking us to "think about problems" is kind of the opposite of what good leadership looks like. Thinking about thinking about problems is like, 200% wrong. On the contrary, be way more emotional and way simpler.
And since this network is privileged, observability tools, debugging support, and even maybe access to it are more complicated. Even just the set of engineers who have access is likely more limited, especially at 2AM.
Should AWS relax these controls to make recovery easier? But then it will also result in a less secure system. It's again a trade-off.
The problem is, oncall is a full-time business. It takes full attention of the oncall engineer, whether there is an issue or not. Both companies simply treat oncall as a by-product. We just had to do it so let’s stuff it into the sprint. The first company was slightly more serious as we were asked to put up a 2-3 point oncall task in JIRA. The second one doesn’t even do this.
Neither company really encourages engineers to read through complex code written by others, even if we do oncall for those products. Again, the first company did better, and we were supposed to create a channel and pull people in, so it’s OKish to not know anything about the code. The second company simply leaves oncall to do whatever they can. Neither company allocates enough time for engineers to read the source code thoroughly. And neither has good documentation for oncall.
I don’t know the culture of AWS. I’d very much want to work in an oncall environment that is serious and encourages learning.
We sent a test page periodically to make sure the pager actually beeped. We got paid extra for being in the rotation. The leadership knew this was a critical step. Unfortunately, much of our tooling was terrible, which would cause false pages, or failed critical operations, all too frequently.
I later worked on SWE teams that didn't take dev oncall very seriously. At my current job, we have an oncall, but it's best effort business hours only.
12-12 rotation in SRE is a lot more reasonable for humans
It was a good lesson in what a manicured lower environment can do for you.
Is that really uncommon? I've been on call for many companies and many types of institutions and never been told once I couldn't do something to bring a system up that I can recall at least. Its kinda the job?
On call seriousness should be directly proportional to pay. Google pays. If smallcorp want to pay me COL I'll be looking at that 2AM ticket at 9AM when I get to work.
That piece by Cook is ok, but largely just a list of assertions (true or not, most do feel intuitive, though). I suppose one should delve into all those references at the end for details? Anyway, this is an ancient topic, and I doubt we have all the answers on those root whys. The MIT course on systems, 6.033, used to assign reading a paper raised on HN only a few times in its history: https://news.ycombinator.com/item?id=10082625 and https://news.ycombinator.com/item?id=16392223 It's from 1962, over 60 years ago, but that is also probably more illuminating/thought provoking than the post mortem. Personally, I suspect it's probably an instance of a https://en.wikipedia.org/wiki/Wicked_problem , but only past a certain scale.
You might have to bring personal trauma to this piece to get the full effect.
In engineered systems, there is just a disconnect between on our own/small scale KISS and what happens in large organizations, and then what happens over time. This is the real root cause/why, but I'm not sure it's fixable. Maybe partly addressable, tho'.
One thing that might give you a moment of worry is both in that Simon and far, far more broadly all over academia both long before and ever since, biological systems like our bodies are an archetypal example of "complex". Besides medical failures, life mostly has this one main trick -- make many copies and if they don't all fail before they, too, can copy then a stable-ish pattern emerges.
Stable populations + "litter size/replication factor" largely imply average failure rates. For most species it is horrific. On the David Attenborough specials they'll play the sad music and tell you X% of these offspring never make it to mating age. The alternative is not the https://en.wikipedia.org/wiki/Gray_goo apocalypse, but the "whatever-that-species-is-biopocalypse". Sorry - it's late and my joke circuits are maybe fritzing. So, both big 'L' and little 'l' life, too, "is on the edge", just structurally.
https://en.wikipedia.org/wiki/Self-organized_criticality (with sand piles and whatnot) used to be a kind of statistical physics hope for a theory of everything of these kinds of phenomena, but it just doesn't get deployed. Things will seem "shallowly critical" but not so upon deeper inspection. So, maybe it's not not a useful enough approximation.
Anyway, good luck with your housing meetup!
1. Complex systems are intrinsically hazardous: AWS’s global cloud is exactly what Cook meant: tightly coupled, high-speed, distributed, opaque. Its "hazard" is downtime—something they spend billions automating away. Yet automation itself (DNS Planner + DNS Enactor) introduced a new hazard class: race conditions across redundant agents. The automation meant to reduce failure probability became a new failure source.
2. Systems are heavily defended—catastrophe requires multiple small failures: This wasn’t one bug. DNS Enactor retry lag + stale plan checks; Planner producing newer generations + cleanup mis-timed; manual recovery blocked by inconsistent Route53 state; DWFM (EC2) dependent on DynamoDB leases -> cascading collapse. Each alone survivable; in combination they crossed the threshold. Exactly Cook’s "innocuous failures joining to create a systemic accident."
3. Systems run in degraded mode: AWS was already operating "broken but functional"—Enactors retrying, Planner issuing new plans, DWFM expiring leases—all silently degraded but masked by redundancy. Only when timing aligned did latent faults synchronize. Normal accidents emerge from normal degraded conditions.
4. Human operators are the adaptable glue: Automation deadlocked; recovery required human override of DNS state, manual DWFM throttling, and ad-hoc re-sequencing of NLB health checks. Operators re-stitched the fabric live—Cook’s "people continuously create safety" in action. Resilience was restored more by improvisation than automation.
5. Post-event "root cause" is an illusion: AWS’s write-up isolates a "race condition in DNS Enactor." Per Cook, that’s a social simplification—an accountability ritual that hides the web of mutual coupling between EC2, Route53, DynamoDB. There was no single root cause—only joint sufficiency of timing, automation design, and dependency.
6. Change introduces new forms of failure: The failure path appeared because AWS introduced IPv6 DNS consolidation and Enactor concurrency to "simplify capacity management." That optimization created an untested timing pathway—the "new, low-frequency, high-impact failure" Cook predicted. Eliminating old complexity birthed a subtler one.
7. Safety is systemic, not component-based: Disabling DNS automation worldwide adds local "protection," but may increase systemic coupling and operator workload. True safety emerges when AWS models cross-service dependency networks holistically (DNS -> EC2 -> Lambda -> Connect, etc.).
8. Practitioners operate on the edge: Operators trade off production vs. safety: automation cadence vs. plan propagation speed, throttling vs. capacity recovery. Their choices are gambles under uncertainty—Cook’s "actions are gambles." In hindsight the Enactor’s logic looks wrong; at runtime it was locally rational.
9. Defensive layering breeds new brittleness: AWS’s multiple redundancy layers (Planner, 3x Enactors, failover AZs) synchronized to produce the deletion. Redundancy without temporal coordination increased failure correlation—a known paradox of highly automated systems.
10. People create safety, not procedures: The outage ended because humans improvised new DNS repair paths and throttling strategies. That’s emergent safety—Cook’s final thesis: "Failure-free operation is the result of people keeping the system within boundaries."
Takeaway: AWS as a living complex system - Intrinsic hazard — Massive scale, dynamic DNS automation - Layers of defense — Multi-AZ redundancy, automation, monitoring - Multiple small failures — Planner + Enactor + cleanup sequence - Degraded operation — Retrying Enactors, DWFM backlog - Human adaptation — Manual DNS repair, DWFM throttling - No single root cause — Cascading race + dependencies - New failure from change — IPv6 + unified plan logic - Safety is emergent — Operator recovery behavior
Doing this isn't helpful.
Edit: Oops I looked at 8, it's also wrong, the Enactor setting the plan wasn't locally rational, it made a clear mistake. Also that claim has nothing to do with the rest of the paragraph! This output is so bad.
One could make a similar argument in sports that no one person ever scores a point because they are only put into scoring position by a complex series of actions which preceded the actual point. I think that's technically true but practically useless. It's good to have a wide perspective of an issue but I see nothing wrong with identifying the crux of a failure like this one.
>>> Views of ‘cause’ limit the effectiveness of defenses against future events.
>>> Post-accident remedies for “human error” are usually predicated on obstructing activities that can “cause” accidents. These end-of-the-chain measures do little to reduce the likelihood of further accidents. In fact that likelihood of an identical accident is already extraordinarily low because the pattern of latent failures changes constantly. Instead of increasing safety, post-accident remedies usually increase the coupling and complexity of the system. This increases the potential number of latent failures and also makes the detection and blocking of accident trajectories more difficult.
For example, take airline safety -- are we to believe based on the quoted assertion that every airline accident and resulting remedy that mitigated the causes have made air travel LESS safe? That sounds objectively, demonstrably false.
Truly complex systems like ecosystems and climate might qualify for this assertion where humans have interfered, often with best intentions, but caused unexpected effects that maybe beyond human capacity control.
But I can think of lots of examples where the response to an unfortunate, but very rare, incident can make us less safe overall. The response to rare vaccine side effects comes immediately to mind.
Then I realized: the internet; the power-grid (at least in most developed countries); there are things that don't actually fail catastrophically, even though they are extremely complex, and not always built by efficient organizations. Whats the retort to this argument?
I think you could argue AWS is more complex than the electrical grid, but even if it's not, the grid has had several decades to iron out kinks and AWS hasn't. AWS also adds a ton of completely new services each year in addition to adding more capacity. E.g. I bet these DNS Enactors have become more numerous and their plans became much larger than when they were first developed, which has greatly increased the odds of experiencing this issue.
This has gotten significantly better in recent years, but it used to be possible and common for a single misbehaving AS to cause global issues.
Texas nearly ran into this during their blackout a few years ago -- their grid got within a few minutes of complete failure that would have required a black start which IIRC has never been done.
Grady has a good explanation and the writeup is interesting reading too.
Pointing out that "complex systems" have "layers of defense" is neither insightful nor useful, it's obvious. Saying that any and all failures in a given complex system lack a root cause is wrong.
Cook uses a lot of words to say not much at all. There's no concrete advise to be taken from How Complex Systems Fail, nothing to change. There's no casualty procedure or post-mortem investigation which would change a single letter of a single word in response to it. It's hot air.
– It identifies problems (complexity, latent failures, hindsight bias, etc.) more than it offers solutions. Readers must seek outside methods to act on these insights.
– It feels abstract, describing general truths applicable to many domains, but requiring translation into domain-specific practices (be it software, aviation, medicine, etc.).
– It leaves out discussion on managing complexity – e.g. principles of simplification, modular design, or quantitative risk assessment – which would help prevent some of the failures it warns about.
– It assumes well-intentioned actors and does not grapple with scenarios where business or political pressures undermine safety – an increasingly pertinent issue in modern industries.
– It does not explicitly warn against misusing its principles (e.g. becoming fatalistic or overconfident in defenses). The nuance that «failures are inevitable but we still must diligently work to minimize them» must come from the reader’s interpretation.
The treatise by itself won’t tell you how to design the next aircraft or run a data center more safely, but it will shape your thinking so you avoid common pitfalls (such as chasing singular root causes or blaming operators). To truly «preclude» failures or mitigate them, one must extend Cook’s ideas with detailed engineering and organizational practices. In other words, Cook teaches us why things fail in complex ways; it is up to us – engineers, managers, regulators, and front-line practitioners – to apply those lessons in how we build and operate the systems under our care.
To be fair, at the time of writing (late 1990's), Cook’s treatise was breaking ground by succinctly articulating these concepts for a broad audience. Its objective was likely to provoke thought and shift paradigms, rather than serve as a handbook.
Today, we have the benefit of two more decades of research and practice in resilience engineering, which builds on Cook’s points. Practitioners now emphasise building resilient systems, not just trying to prevent failure outright. They use Cook’s insights as rationale for things such as chaos engineering, better incident response, and continuous learning cultures.
https://how.complexsystems.fail/
You can literally check off the things from Cook's piece that apply directly here. Also: when I wrote this comment, most of the thread was about root-causing the DNS thing that happened, which I don't think is the big story behind this outage. (Cook rejects the whole idea of a "root cause", and I'm pretty sure he's dead on right about why.)