Image you have two servers, each with two 1TB disks (4TB physical storage). And you have two distinct services with 1TB datasets, and want some storage redundancy.

One option is to put each pair of discs in a RAID-1 configuration, and so each RAID instance holds one dataset. This protects against a disk failure, but not against server or network failures.

Your other option is to put one copy of the dataset on each server. Now you are protected from the failure of any single disk, server, or the network to that server.

In both cases you have 2TB of logical storage available.

In either case what you're describing isn't really the 'cloud' alternative.

The parents point is really spot on though: most websites aren't at scale where every stateful api has redundant implementations. But the author's point does have merit: inevitably, something goes wrong with all systems - and when it does, your system goes down if you trust your HA config to work. If you actually did go for redundant implementations, your users likely aren't even gonna notice anything went wrong.

It's however kinda unsustainable to maintain unless you have a massive development budget

He's also doing wasted work his redundancy has bugs that a consumer has to worry about working around.

I think the author was specifically talking about RAID-1 redundancy and is advocating that you can leave your systems with RAID-0 (so no redundant drives in each server), as you're gonna need multiple nodes in your cluster anyway... so if any of your systems disks break, you can just let it go down and replace the disk while the node is offline.

but despite being offtopic: redundant implementations are -from my experience - not used in a failover way. they're active at all times and load is spread if you can do that, so you'd likely find the inconsistencies in the integration test layer.

Aurora works like that. Read replicas are on standby and become writable when the writable node dies or is replaced. They can be used, of course, but so can other standby streaming replicas.

The larger your infrastructure, the smaller the relative efficiency win that's worth pursuing (duh, I know, engineering time costs the same, but the absolute savings numbers from relative wins go up). That's why an approach along the lines of "redundancy at all levels" (raid + x-machine replication + x-geo replication etc) starts becoming increasingly worth streamlining.

Another, separate consideration is types of failures you have to consider: an availability incident (temporary unavailability) vs. durability (permanent data loss). And then it's worth considering that in the limit to long durations, an availability incident will become the same as a durability incident. This is contextual: To pick an obvious/illustrative example, if your Snapchat messages are offline for 24h, you might as well have lost the data instead.

Now, machines fail, of course. Doing physical maintenance (swapping disks) is going to take significant, human time scales. It's not generally tolerable for your data to be offline for that long. So local RAID barely helps at all. Instead, you're going to want to make sure your data stays available despite a certain rate of machine failure.

You can now make similar considerations for different, larger domains. Network, power, building, city/location, etc. They have vastly different failure probabilities and also different failure modes (network devices failing is likely an availability concern, a mudslide into your DC is a bit less likely to recover). Depending on your needs, you might accept some of these but not others.

The most trivial way to deal with this is to simply make sure you have a replica of each chunk of data in multiple of each of these kinds of failure zones. A replica each on multiple machines (pick the amount of redundancy you need based on a statistical model from component failure rates), a replica each on machines under different network devices, on different power, in different geographies, etc.

That's expensive. The next most efficient thing would be to use the same concept as RAID (erasure codes) and apply that across a wider scope. So you basically get RAID, but you use your clever model of failure zones for placement.

This gets a bit complicated in practice. Most folks stick to replicas. (Eg. last I looked, for example HDFS/Hadoop only supported replication, but it did use knowledge of network topology for placing data.)

The reason why you don't want to do this in your application is because it's really kinda complicated. You're far more likely to have many applications than many storage technologies (or databases).

Now, at some point of infrastructure size or complexity or team size it may make sense to separate your storage (the stuff I'm talking about) from your databases as well. But as Kris argues, many common databases can be made to handle some of these failure zones.

In any case, that's the extremely long version of an answer to your question why you'd handle redundancy in this particular layer. The short answer is: Below this layer is too small a scope or with too little meta information. But doing it higher in the stack fails to exploit a prime opportunity to abstract away some really significant complexity. I think we all know how useful good encapsulation can be! You avoid doing this in multiple places simply because it's expensive.

(Everything above is common knowledge among storage folks, nothing is Google specific or otherwise it has been covered in published articles. Alas, the way we would defend against bugs in storage is not public. Sorry.)

That provides capacity, but also redundancy. Better redundancy than at the disk level - fewer resources are shared.

https://blog.koehntopp.info/2021/03/24/a-lot-of-mysql.html Here is how we run our datatbases.

That's one of the perceptions I'm saying isn't always true. Especially in big, non-tech companies that have a mish-mash of crazy legacy stuff. Traditional H/A disks and backups still dominate some spaces.

Only really huge services would care about downtime of this degree, or horizontal scaling.

At 400 MB/s, you restore a terabyte in 45 minutes, and then you need to replay the changes that happened since the backup (in MySQL: replay the binlog), which will take aproximately another 15 minutes.

"One hour of MTTR or provisioning time per Terabyte of data at 400 MB/s sustained linear I/O speed" is a useful rule of thumb.

Having a replica that you can promote reduces that to seconds. Having a time delayed replica where you just roll forward the binlog reduces that to minutes.

You can work with modern databases without using replication, but that in most cases just shows you suck at operations.

When I ran large-scale Exchange and database systems, we would always get into big fights with the storage people, who believed that SAN disk with performance tiering, replication, etc was the way to solve all problems.

The problem at the time was that the fancy storage cost $40+ Gb/mo and performed like hot garbage because the data didn’t tier well. For email, the “correct” answer in 2010 was local disk or dumb sas arrays with exchange dag.

For databases, the answer was “it depends”. Often it made sense to use the SAN and replication at that layer.

it is not. if you have 10 DB hosts with the same brand of nvme, and they fail under a workload, what good is it that you have 10 hot-hot failover hosts? you just bought yourself a few days, or hours if you are specially unlucky and using consumer grade.

For that you have time delayed replicas and of course backups. But time delayed replicas are usually much faster to restore than a backup.