We would containerize everything if we could! But our infrastructure components usually live outside the security boundaries our container interface sets up. The proxy has to be able to talk to every app, not just apps in its own organization, and it needs direct access to other infrastructure components. The orchestration code needs to be able to launch Firecracker VMs --- it can't itself be a Firecracker VM.

We can design around all these constraints! We just haven't gotten around to it yet. I think there's a general consensus that hiding things inside Firecrackers is a good design, and we'll do it where we can.

Elixir runs great inside Firecrackers. The idea behind Firecracker is that it introduces a minimal load, by dint of being ruthlessly simple. When you run an Elixir app on an EC2 instance, you're running hypervised as well (and probably by a hypervisor that's less efficient --- not a knock, Firecracker is Amazon code too).

The thing that makes this doable at Fly.io is that we run our own hardware, so we're not trying to nest VMs inside VMs. Of course, that's what EC2 is doing too. So whatever your intuitions are about things that run well on vanilla EC2, they should carry over to us.

What are you using for the host OS on your bare metal?

Linux. :)

heh, could you share if it's more of an rpm flavor or more of a deb flavor? :-) (or some third option. no false dichotomies intended. actually it would be badass if you're running Arch :-D)

At this moment I will go as far as to say that it is not a badass flavor of Linux.

:)