I'd think the key on this is being able to simulate very specific partial-failure conditions. e.g. specific packet loss, loss of connections to EBS, etc. Just turning machines off I expect wouldn't be that valuable.

If Amazon simulated power failure, or network cable disconnect, or potentially even corrupt writes to disk, I personally feel it would be indistinguishable from the event really happening.

echo c > /proc/sysrq-trigger

A panicking instance is still "alive" from its hypervisor's perspective (either it'll hang, sleep, or reboot, but it won't usually turn off its vCPU in a way the hypervisor would register as "the instance is now off"); while if a hypervisor box suffers a power cut, the rest of the compute cluster knows that the instances on that node are now very certainly off.

https://www.kernel.org/doc/html/v4.11/admin-guide/sysrq.html...

Looks like SysRq-o does a clean poweroff, not a dirty/immediate one — it calls kernel_power_off(), not machine_power_off().

This means, importantly, that can actually take some time to happen, as drivers get a chance to run deinitialization code. This also means that a wedged driver that doesn't respond during deinit can prevent the kernel from halting.

Thus, while SysRq-o might be useful for killing a wedged userland, it's not a panacea — especially, it isn't guaranteed to complete a shutdown for unstable kernels, or kernels with badly-written DKMS drivers attached. It's not truly equivalent to a power-cut.

Granted, most of those are out of scope for cloud development, but the concept of externally cutting off a VM is different than even calling whatever power service you have to cut power. In the 'real' world, enough of those failure triggers above would probably also trigger an automated power cycle if you're in a managed environment.