News for hackers. Stuff that matters.
Re:NORAD's Cheyenne Mountain Combat Center, C.1966 by Lucian6 on Sunday October 26, 2025 @11:23PM (#45717103)
The redundancy systems in Cheyenne Mountain are fascinating from a modern distributed systems perspective. When designing our real-time services, we often struggle with 4-9s uptime, yet NORAD achieved remarkable reliability with 1960s technology. Their approach of physical isolation combined with multiple independent power/cooling/compute systems is still relevant today.

One detail that caught my attention: they used triple-redundant IBM 7090 mainframes with real-time voting between systems. This is surprisingly similar to modern consensus protocols like Raft, just implemented in hardware. Makes me wonder if we've over-complicated things with software-based consensus when simple majority voting could suffice for many use cases. The key seems to be truly independent failure domains - something NORAD achieved through physical separation that's harder to replicate in cloud environments.

Would love to hear from others who've implemented hardware-based consensus systems. How do they compare to software solutions in terms of reliability and complexity?

Re:NORAD's Cheyenne Mountain Combat Center, C.1966 by toast0 on Monday October 27, 2025 @12:19AM (#45717346)
> The key seems to be truly independent failure domains - something NORAD achieved through physical separation that's harder to replicate in cloud environments.

It's not hard to achieve separation in cloud environments. It's just expensive, and for most, the need doesn't justify the expense. Every once in a while you do run into things like actually all the clouds in (location) are located in the same building; or you paid for two separate strands, but a backhoe has determined that both strands are in the same bundle. Or your two redundant providers of whatever use each other to provide service.

Re:NORAD's Cheyenne Mountain Combat Center, C.1966 by kqr on Monday October 27, 2025 @02:10AM (#45717879)
I'm always in favour of simplifying things, and in our case complexity breeds further complexity. We are making software recursively, building software on software on software on software, and this has more inherent complexity than just hardware. Why do we do this? It allows anyone to sit in their bedroom and build something in no time at all, compared to the large groups of people with specialised, expensive hardware it would require in the 1960s.

With more complexity comes more interesting failure scenarios. Simple redundancy handles problems where components outright fail, but software never really ... fails, in that sense. It always performs exactly to design – but sometimes the design is inadequate. Replication and majority voting does not handle design flaws and errors of e.g. interaction, which are increasingly common the more softwareified a product is.

Thus we need more complicated ways to build fault tolerance into systems where we have to assume that the design is wrong somewhere, and we cannot just replicate ourselves out of the problem.