I'm sure Amazon would love it too (think about how much money they could save / make) but there are a lot of security concerns that need to be addressed. Running code in a virtualized kernel is a lot more secure than running in the real kernel with permissions limited. Theoretically an exploit could allow someone to take over the machine, whereas if the machine is virtualized it's much harder to take over the host.

Doesn't matter if the machine is taken over if its properly contained at the network level. Dedicated server hosting companies have been doing this for years.

With VMs the customer can share a physical host with others and not have security concerns. This cohosting lets aws pack customers more efficiently and have much better margins for the same quality service.

With containers on bare metal, they could not cohost customers.

Uhh, even with containers you really can't cohost in a multi-tenant system. Containerization doesn't securely segregate arbitrary code in the same way traditional virtualization does, and there are still security issues with that.

I'm sure AWS is improving on this, as they need to with Lambda and its underlying containerization architecture. But everyone else? Its going to be a while before you can call it "secure".

It looks like you are in violent agreement

We are! And I love that phrase :)

Amazon offers billing by the hour for those, meaning they probably move between customers a lot more than traditional dedicated servers. A bare-metal user can do lot of bad things unless you trust your hardware manufacturing 100 %, e.g. by messing with the firmware.

At the hardware purchasing level Amazon operates with for AWS hardware, I'm sure they're able to (if they required it) spec custom BIOS requirements that perform integrity verification of subsystems before permitting power up to continue. Whether they do or not is a different question (although I'm sure someone could test this theory).

They could also just spec that firmware be stored on ROM rather than flash.

Or that a hardware jumper needs to be set on the motherboard before any firmware updates can be done.

Rackspace offers a cloud cum bare metal service called OnMetal. They flash the firmware before putting a new user on a machine.

Unless “the machine is taken over” means firmware-level compromise (e.g. disk controller, NIC controller, …), in which case the next customer allocated to that machine can be compromised despite full disk erase between customers.

I'll agree with this attack vector, although it'd be trivial to have your PXE boot routine between customers re-apply known good firmware to the NIC and disk controller and verify with checksums (unless there's a 0-day out there for the NIC controller, in which case you're owned regardless).

Normally the PXE boot code is, itself, stored in firmware.

Whatever components start first, wins. If the PXE boot code is in the NIC, you can theoretically verify the rest of the system first. If its in the BIOS, the BIOS needs to trust the NIC as untrusted/determine if its compromised or not.

Security is hard :/

ROM based NIC? Why not? If you're going into large scale dedicated servers, this is not exactly impractical. You could probably also just disconnect the flash enable pins on the ICs.

Assuming the firmware has no say in the flashing process.

It definitely does matter if the host machine is taken over... an attacker could access other customer instances, shut down the system to try and shut down EC2, etc..

If I've paid for the bare metal machine then by definition I already have root on the host machine. Securing that is just as much my problem at that point as is securing my EC2 instances.

The security concern you mention (taking over the host) is moot if I've paid for root access to a physical machine (bare metal). By definition I've already paid for the host; there's nothing else to break out of.

SDN could easily be handled by an off-system component. I forget who, but someone who presented at ONS 2015 [1] mentioned the use of FPGAs for this.

[1] https://www.youtube.com/playlist?list=PLhigroIsbIuet0qlIGBQi...

I have run containers on bare metal and in Amazon. Performance can be noticeably slower in Amazon on machines with similar specifications. A detailed explanation why is in this informative video by Bryan Cantrill https://www.youtube.com/watch?v=coFIEH3vXPw. It is a talk about how layering containers on top of virtual machines is a recipe for poor performance and wasted resource utilization. He equates it to developers “being given fire, then proceeding to put the fire out.”

so would you say then that the next logical step would be to try out joyent baremetal container runtime?

You might want take a look at SoftLayer, they have bare metal servers. However, if you are okay with ARM servers, then I'd suggest taking a look at scaleway, which provide bare metal ARM servers, and they manage it all for you as well.

If you want bare metal for performance, you're not going to get it on a tiny ARM chip

You can get managed Docker containers on bare metal with Bluemix which runs on Softlayer.

> Running ECS is cool and all, and I know they do some container-specific VM optimizations, but I still know I'm running a kernel on a kernel, even if my VMs happen to share a host.

The cloud is great for distributed systems, and for these the "kernel on a kernel" cost becomes insignificant with the additional power allowed by the horizontal scalability of letting multiple computers talk over a network (not to mention network latency).

I'd be willing to bet that running on absolute bare metal is a sort of niche market in cloud computing that Amazon considers too small to get into.

"kernel on a kernel" is one of the worst things you can do to a distributed computing environment. This is why Google does containers.

I have double digit thousands of machines for one application, and have an Excel spreadsheet outlining the additional hardware that would need to be purchased to handle the same workload but virtualized. I pull it out every time some recent Stanford grad tells me the cloud is the future.

Would you mind explaining a little about how bare metal is different from this? Isn't a dedicated host a real physical machine?

You have a dedicated host to run VM but you don't really have access to the host OS and cannot run custom code on it.

Previously, your VM were allocated to a host as Amazon saw fit.

This is giving you control of the allocation on one specific VM host. So you have, for example, a box that is hosting x m4.larges that you can fill how you want. Plus you have information about the hardware, so software that's licensed per physical machine can be run on each of those instances.

No I think it's still virtualised isn't it? But on a machine that they guarantee only you are using?

Yes. The key is that you can control exactly what combination of your own VMs land on each physical machine.

Agreed. But maybe that's a later step in the roadmap. Let's hope!

What do you mean bare metal? Does that mean physical server?