OS-level 'turn off' options don't replicate what happens when you yank power on a rack of equipment.

Pretty much every option you have from the OS will let caches flush, will have in-progress writes complete.

Yank the power and none of that will happen. It'll let you actually see what level of fibbing your OS and hardware are telling you.

Oh you got a result from that flush to disk? It completed? Are you sure? Really really sure? Lets find out...

The main result of this testing is that you'll find bugs in all the abstraction layers that you can't control. You'll then be paranoid but unable to take any action. Have you ever seen the source code that's running on your SSD's CPU? Nope. And it's probably not a small program. My guess is that it works well when everything is OK, and fails catastrophically 0.0001% of the time when everything isn't OK. But you'll never know unless you try failing catastrophically a million times. Did the vendor even try failing catastrophically one million times before they started manufacturing (or at least sent the batch to retailers)? Did they do that, fail 1 time, and mark the bug closed as non-reproducible?

I have no idea! Maybe everything is actually great. Or maybe someone else will be oncall the week you hit that one in a million failure case. Or maybe it will happen to your competitors instead of you! Without testing, all we have is hope.

> You'll then be paranoid but unable to take any action.

I disagree.

While you can't control the bugs in the hardware or drivers or whatever, you can definitely ensure that your application detects corrupted data and can at least warn that something is wrong rather than starting back up and silently serving from corrupted data.

It also ensures that a node suddenly disappearing without a graceful "I'm leaving" message, and potentially with open transactions or other operations can be handled by the clients and nodes that ARE still around.

You've hit on one of my big pet peeves -- I am going to add redundancy and erasure codes at the application layer because the disk could catch on fire or be sucked up by a tornado, so why do I need a bunch of unaudited software adding extra redundancy and erasure codes behind my back?

I get it for desktop users... they want to open some benchmark they downloaded and get a higher number than their friends (RAID-0-esque sharding data across individual flash chips), and they also don't have access to software that can add error correction codes (because the Windows installer can't recognize an SSD with 4 flash chips as 4 drives in a RAID array)... so the disk has to do it itself. But why is it a thing in datacenters? Just so you can install Linux+ext2+MySQL and call it a day? That seems crazy to me when better storage software exists.

Aren't there SSDs that go in the opposite direction and expose a key-value store instead of a linear address space? Seems like that will make reliability testing even harder.

Even with testing, it’s still just hope; to quote Buzz Lightyear, it’s hoping with style.

I think this is too pessimistic. If all you do is test your error detection and backup recovery mechanism (of course you have backups right)? Then that’s an advantage.

Let’s say a hard power off causes data corruption that can’t be fixed. For lots of applications downtime is better than corruption, so in that case you at least will be able to test when you should take the system down and recover from a known good backup.

kill -9 on the VM process from the host?

If you're using Docker, yes, you can get a lot more options in testing by poking "from the outside", but that still doesn't test what happens when your docker host just dies.

That's not a dumb question at all.

I'm not the GP, but I assume it is because the GP would like to run his "production" setup (or even production setup itself) under such circumstances. So far as I remember, the origin of the idea came from the Chaos Monkey which would sabotage certain services in a production environment to ensure than everything was redundant and fail safe.