Most servers have some sort of "lights out" management, which gives KVM + remote imaging and bios control.

With amazon, they have complete control over the network in and out, so cutting you off and re-imaging a server is pretty trivial.

To be fair, its not that hard to do even if you're not amazon.

Most of the big server vendor's out of band interfaces have an API, so telling a server to reboot from a network image is pretty trivial. Providing a netboot infrastructure to install images with a 'userdata' script is also not that difficult.

you'll need a DHCP server, tftp to serve the boot image, and usuaally an NFS server to pull the rest of the image over. With some engineering work that could be made to use HTTP.

https://wiki.centos.org/HowTos/NetworkInstallServer

It's a bit harder if you host something like this for the general public to use (vs administrating machines in your private DC). Normal setups aren't really hardened against someone flashing firmware, messing with UEFI, ..., all of which mean you can't entirely trust a machine coming back from customer control. I wouldn't be surprised if Amazon took this seriously and invested effort in stopping such things. At their scale, they probably can customize the hardware enough.

Everyone who sells bare metal as a service takes this seriously. As AWS build their own hardware, especially in these newer machines, I would guess that its not possible to flash firmware from the user machine, only from the control node.

EC2 Bare Metal instances boot from an EBS volume that is accessed via a NVMe PCI device (implemented in ASICs built by Annapurna Labs), just like virtualized C5 instances.

Why would you boot what is essentially an iSCSI target via NVMe?

NVMe is just how the storage is surfaced -- the hardware programming interface for the block device. Hardware iSCSI initiators (HBAs) also have a hardware programming interface, but at the end of the day you talk SCSI over that interface.

NVMe is a better match for the the storage operations supported by EBS. A bonus is that by surfacing EBS over NVMe there is a common storage interface for both managed storage volumes and local NVMe storage.

so is that a hardware cache, software cache or a storage tier?

The NVMe interface to EBS volumes that is implemented in our Nitro system today is closest to a HBA with no additional caching. So, a storage tier.

Maybe they use Intel's Management Engine!

The E5-2686 v4 doesn’t have IME.

The CPU itself doesn't, but the chipset does.

Amazon is a big enough customer that it wouldn't surprise me if they could get Intel to make special ME firmware with the features they want in it.

...which also means the ME exploits discussed recently here could lead to a whole lot more fun... ;-)