Yeah, but did he win the Putnam?

2007 HN was highly entertaining

https://en.wikipedia.org/wiki/William_Lowell_Putnam_Mathemat...

I don't pay much heed to AWS Heroes. My city has an "AWS Hero" who hoarded the organization of the official AWS meetup for themself, and who is infamous for making liberal use of Cunningham's Law [1] for everything.

[1] https://meta.wikimedia.org/wiki/Cunningham%27s_Law

Honestly, I don't either. There's a handful of us who are technical and a huge number whose contributions are primarily marketing.

[1] https://localstack.cloud/

The mere existence of such an api would be an interesting source of problems when used accidentally/via buggy code ...

I wonder the degree to which this functionality would end up becoming relied on as a "in worst case hard kill things to recover" behavior that folks utilize for bad engineering reasons as opposed to good ones ...

In a previous job we actually had to make a decision like this: our server software was generally stable enough, but we reset it whenever we did an update. Every once in a while it would break when running for long enough.

I opted for forcefully restarting the software every day, because I'd rather relive the pains of restarting, than to discover in production what it is like for the first time to run the system for the 501st day of uptime.

(To be honest, the system in question was mostly written in Erlang, so controlled restarts were already baked into its philosophy.)

Edit: Another term is Switched PDU

This would be more effective if so many other AWS features weren't hidden in non-obvious places in the docs.

On top of keeping people who don't want it out, you'd need it to be expensive to offset the lower utilization.

I'd be okay with a garbage-tier, too. It's not failing on purpose, but only because no one gives a shit. Maybe that's just standard AWS?

Wouldn't just running "halt -f" do the same?

I'd think the key on this is being able to simulate very specific partial-failure conditions. e.g. specific packet loss, loss of connections to EBS, etc. Just turning machines off I expect wouldn't be that valuable.

If Amazon simulated power failure, or network cable disconnect, or potentially even corrupt writes to disk, I personally feel it would be indistinguishable from the event really happening.

echo c > /proc/sysrq-trigger

A panicking instance is still "alive" from its hypervisor's perspective (either it'll hang, sleep, or reboot, but it won't usually turn off its vCPU in a way the hypervisor would register as "the instance is now off"); while if a hypervisor box suffers a power cut, the rest of the compute cluster knows that the instances on that node are now very certainly off.

https://www.kernel.org/doc/html/v4.11/admin-guide/sysrq.html...

Looks like SysRq-o does a clean poweroff, not a dirty/immediate one — it calls kernel_power_off(), not machine_power_off().

This means, importantly, that can actually take some time to happen, as drivers get a chance to run deinitialization code. This also means that a wedged driver that doesn't respond during deinit can prevent the kernel from halting.

Thus, while SysRq-o might be useful for killing a wedged userland, it's not a panacea — especially, it isn't guaranteed to complete a shutdown for unstable kernels, or kernels with badly-written DKMS drivers attached. It's not truly equivalent to a power-cut.

Granted, most of those are out of scope for cloud development, but the concept of externally cutting off a VM is different than even calling whatever power service you have to cut power. In the 'real' world, enough of those failure triggers above would probably also trigger an automated power cycle if you're in a managed environment.

Even killing a VM instantaneously on a host only simulates the loss of the guest OS's cache. The host OS still has its cache.

And even killing the host OS only simulates loss of the host OS's cache. The drives still have their caches powered, so might behave differently on power loss than host OS crash.

And killing a drive while the host keeps running is different again.

(And in terms of high-level software-observable data effects, abruptly killing power to a drive is not the same as slowly lowering the voltage or limiting the current to the drive, or seeing corrupt data on the I/O bus while the host system loses power (which is never instant), or... you can go quite deep with this.)

In the networked storage environment at AWS, who knows what events are possible to observe on EBS in the event of a system-level failure such as power loss in a data centre.

For example, EBS is a distributed system with replication, so if it has an implementation error that only affects certain untested, complex failure modes, it might lose recent writes in flight (as expected), but then some time later they might come back, or some of them might come back if it's sharded. Both are bad if software using it has already started state recovery after an outage. Distributed systems often have surprising incorrect recovery patterns, because it's a complex and subtle problem; that's why the Jepsen tests keep finding bugs in databases people have been using for years.

The observable events can differ depending on whether it's VM hosts, switches, block storage units, drives failing and in what ways.

For Linux guests we have abstracted away most differences we care about under fsync() or various combinations of O_DIRECT and virtual HDD cache disabling, or relying on known features of filesystems, but there's no easy way to be certain those abstractions actually provide the expected semantics on different kinds of system failure. Stress testing higher layers in the virtualization stack would go some way to verifying that they do in practice, not just in theory.

To my mind simply turning off the networking, physically cutting a cable, or hardware spontaneously combusting are not discernible events to an outside observer. What am I missing?

Pretty much every option you have from the OS will let caches flush, will have in-progress writes complete.

Yank the power and none of that will happen. It'll let you actually see what level of fibbing your OS and hardware are telling you.

Oh you got a result from that flush to disk? It completed? Are you sure? Really really sure? Lets find out...

I have no idea! Maybe everything is actually great. Or maybe someone else will be oncall the week you hit that one in a million failure case. Or maybe it will happen to your competitors instead of you! Without testing, all we have is hope.

I disagree.

While you can't control the bugs in the hardware or drivers or whatever, you can definitely ensure that your application detects corrupted data and can at least warn that something is wrong rather than starting back up and silently serving from corrupted data.

It also ensures that a node suddenly disappearing without a graceful "I'm leaving" message, and potentially with open transactions or other operations can be handled by the clients and nodes that ARE still around.

I get it for desktop users... they want to open some benchmark they downloaded and get a higher number than their friends (RAID-0-esque sharding data across individual flash chips), and they also don't have access to software that can add error correction codes (because the Windows installer can't recognize an SSD with 4 flash chips as 4 drives in a RAID array)... so the disk has to do it itself. But why is it a thing in datacenters? Just so you can install Linux+ext2+MySQL and call it a day? That seems crazy to me when better storage software exists.

Let’s say a hard power off causes data corruption that can’t be fixed. For lots of applications downtime is better than corruption, so in that case you at least will be able to test when you should take the system down and recover from a known good backup.

I'm not the GP, but I assume it is because the GP would like to run his "production" setup (or even production setup itself) under such circumstances. So far as I remember, the origin of the idea came from the Chaos Monkey which would sabotage certain services in a production environment to ensure than everything was redundant and fail safe.

  # cat > dropme.sh <<'EOFILE'
  #!/bin/sh
  set -eu
  read -r SLEEP
  tmp=`mktemp` ; tmp6=`mktemp`
  iptables-save > $tmp
  ip6tables-save > $tmp6
  for t in iptables ip6tables ; do 
    for c in INPUT OUTPUT FORWARD ; do $t -P $c DROP ; done
    $t -t nat -F ; $t -t mangle -F ; $t -F ; $t -X
  done
  sleep "$SLEEP"
  iptables-restore < $tmp
  ip6tables-restore < $tmp6
  rm -f $tmp $tmp6
  EOFILE
  # chmod 755 dropme.sh
  # ncat -k -l -c ./dropme.sh $(ifconfig eth0 | grep 'inet ' | awk '{print $2}') 12345 &
  # echo "60" | ncat -v $(ifconfig eth0 | grep 'inet ' | awk '{print $2}') 12345

https://en.wikipedia.org/wiki/Plurale_tantum

That is, it seems to be grammatically plural but not have a distinctive singular.

     4. pl. A piece of real estate; a building and its adjuncts;
        as, to lease premises; to trespass on another's premises.
        [1913 Webster]

And one day there will be PETA [0] for EC2 instances!

[0]: https://www.peta.org/