“Big cloud” has had fires take out clusters, and somehow they manage to keep it out of the news. In spite of the redundancy and failover procedures, keeping your data centers running when one of the clusters was recently *on fire* is something that is often only possible due to heroic efforts.

When I say “heroic efforts”, that’s in contrast to “ordinary error recovery and failover”, which is the way you’d want to handle a DC fire, because DC fires happen often enough.

The thing is, while these big companies have a much larger base of expertise to draw on and simply more staff time to throw at problems, there are factors which incentivize these employees to *increase risk* rather than reduce it.

These big companies put pressure on all their engineers to figure out ways to drive down costs. So, while a big cloud provider won’t make a rookie mistake—they won’t forget to run disaster recovery drills, they won’t forget to make backups and run test restores—they *will* do a bunch of calculations to figure out how close to disaster they can run in order to save money. The real disaster will then reveal some false, hidden assumption in their error recovery models.

Or in other words, the big companies solve all the easy problems and then create new, hard problems.

I'm curious what references or leads I might follow to learn more about these fires and other events you mention.

Get a job working at these companies and go out for drinks with the old-timers.

You know, those are excellent observations. But they don’t change the decision calculus in this case. Using bigger cloud providers doesn’t eliminate all risk, it just creates a different kind of risk.

What we call “progress” in humanity is just putting our best efforts into reducing or eliminating the problems we know how to solve without realizing the problems they may create further down the line. The only way to know for sure is to try it, see how it goes, and then re-evaluate later.

California had issues with many forest fires. They put out all fires. Turns out, that solution creates a bigger problem down the line with humongous uncontrollable fires which would not have happened if the smaller fires had not been put out so frequently. Oops.

I encourage you to have a look at the operating income that AWS rakes in.

Sure, the amount of expertise, redundancy and breadth of service offerings they provide is worth a markup, but they are also significantly more expensive than they need to be.

Thanks to being the leader in an oligopoly, and due to patterns like making network egress unjustifiably expensive to keep you (/your data) from leaving.

I think the question here, then is of subjective value.

AWS may charge more for egress, but that’s not high enough for it to be a concern for most clients.

A bigger, independent concern is probably that there should be sufficient redundancy, backups and such that allows for business continuity. (Note again that I’m not saying that all companies make full use of these features, but those that care for such things do. Additionally, I’ve honestly never heard of an AWS DC burning down. Either it doesn’t happen frequently or it doesn’t have enough of an effect on regular customers, both of situations are equivalent for my case).

Most businesses choose to prioritize the second aspect. Even if they have to pay extra for egress sometimes, it’s just not big enough of a concern as compared to businesses continuity.

An availability zone (AZ) in AWS eu-west-2 was flooded by a fire protection system going off within the last year. It absolutely did affect workloads in that AZ. That shouldn't have had a large impact on their customers since AWS promote and make as trivial as is viable multi-AZ architectures.

Put another way: one is guided towards making operational good choices rather than being left to discover them yourself. This is a value proposition of public clouds since it commoditises that specialist knowledge.

What surprised me most about today's fire is that their datacenters have so little physical separation. I expected them to be far enough apart to act as separate availability zones.

Hm, I can't find anything in google about this flooding incident. Can you share some details / source?

I've never heard of any data centre burning down (and I work in this industry), so never hearing of an AWS DC burning down isn't really saying anything about AWS.

I remember hosting.ua DC mysteriously "catch fire"

AWS and GCP are also prone to same kind of data loss if the AZ you are operating in goes down.

They don't automatically geo-replicate things. You still need a backup for the torched EC2 instance to be able to relaunch in another AZ/region.

That's true, but it seems whole of SBG region for OVH is within same disaster radius for one fire... with SBG2 destroyed and SBG1 partly damaged.

"The whole site has been isolated, which impacts all our services on SBG1, SBG2, SBG3 and SBG4. "

Wonder if those SBGx were advertised as being the same as "Availability Zones" - when other cloud providers ensure zones are distanced enough from each other (~1km at least) to likely survive events such as fire.

They were not and never advertised as anything similar to AZ. You could not deploy in SBG1,2,3, etc. You only pick city = Strasbourg at deploy time. It's merely a building marker.

> but it seems whole of SBG region for OVH is within same disaster radius for one fire

SBG is for Strasbourg. That's not a region. It's a city. Obviously, SBG1 to 4 are in the radius of one fire. It's four different buildings on the same site.

The buildings are VERY close to one another.

https://cdn.baxtel.com/data-center/ovh-strasbourg-campus/pho...

That seems, uh, problematic.

Why? They aren't making any sort guarantees with those locations, those aren't advertised as separated fault zones. Those buildings are meant as expansions for existing site.

Thats a fair point. If OVH does market them as AZs then it's disingenuous and liable to suits IMO.

No, it isn't, as there's no clear cut definition of what an availability zone is.

If the data is that critical, surely you would be backing it up frequently and also mirror it on at least one geographically separate server?

I use a single server at OVH, and I'm not in the affected DC, but if this DID happen to me I could get back up and running fairly quickly. All our data is mirrored on S3 and off site backups are made frequently enough it wouldn't be an issue.

Plus, you still need to plan for a scenario like this even with AWS or any other cloud provider. It is less likely to happen with those, given the redundancy, but there is still a chance you lose it all without a backup plan.

Yup, I've never heard of a fire taking out a Big Cloud DC. They actually know what they're doing and don't put server racks in shipping containers stacked on top of each other. If you want quality in life, sometimes you have to pay for it.

Personally I'll continue to use these third world cloud providers. But I like to live on the edge.

Apple fire: https://www.datacenterdynamics.com/en/news/fire-rages-throug...

Google fire: https://www.google.nl/amp/s/gigazine.net/amp/en/20060313_goo...

AWS fire: https://money.cnn.com/2015/01/09/technology/amazon-data-cent...

> Apple fire

Their solar panels on the roof caught on fire. Presumably there was no actual disruption in service computing-wise.

> Google fire

2006, still a young company at that point. Also this was way before GCP where they were responsible for other people's services.

> AWS fire

Data center caught on fire while it was under construction. Drunk construction workers messed up, not AWS. Presumably no computers were even inside yet. This is completely different from OVH completely burning down while all their customers' data went up in flames.

If these are the best examples of Big Cloud messing up, this is quite the endorsement.

It's funny how the OVH CEO tells everyone to go activate their DRPs while the company didn't have enough foresight to install fire suppression systems.

These are just the first examples I could find. Fires account for 20%+ of all DC outages in the industry. But taking the time to find a reason to invalidate each one, it seems like you have your mind set.