Any take on how this looks for someone who doesn't work at Amazon?

I know Apollo is their deployment tool. No idea how they plays into a Kubernetes or even container stack.

Conceptually, there is a lot of overlap. Apollo had a few layers to it but would, more or less, matched kubernetes plus additional tooling. Dare I say closer to openshift? Brazil, the build tool, infrastructure and repository was similar to using nixpkgs. Brazil had some features beyond nixpkgs that were absolutely essential for an business like Amazon, but otherwise the goals and even techniques overlapped.

I onboarded very, very quickly by already knowing nixpkgs and container management. The concepts translated well. A few aspects that didn't either were things Amazon handled that other tools have yet to encounter: Cool to learn about. Or historical bits that didn't matter: fine to ignore.

> Apollo had a few layers to it but would, more or less, matched kubernetes plus additional tooling

Have you looked at the codebase? Apollo and Brazil are tiny compared to Kubernetes & friends. You could reimplement them with a small script on top of traditional Linux package management (modulo the web UI).

As someone who worked on Apollo, I'd challenge this. Without divulging any details, Apollo is a non trivial system.

Yeah, I work adjacent to the Apollo org (and have worked near Pfrheak, too. Hi!). I would love to see a HackerNews commenter's idea of what it would take to re-implement Apollo. I'll bring popcorn.

> you would want to use bare metal for Amazon's scale

Containers are "bare metal".

> Containers are "bare metal"

Please explain this use of jargon. Are you saying that a Docker container is "bare metal" in some culture? A VM is bare metal? What do you mean by container, and bare metal?

Containers are just namespaces for things within the Linux kernel. Unlike with VMs, you're not running separate instances of the OS, it's all run by just one kernel instance, and that kernel usually runs directly on the hardware, that is, on "bare metal". That "ubuntu" base image you can spin up does not actually run the Ubuntu kernel. As a result, bare metal containers incur none of the "virtualization penalty" that VMs do.

Cloud is, in fact, abnormal in terms of how containers are used in that it wraps them into VMs. That's in part because VMs were already there and they simplify provisioning of something that looks like "machines" from the outside, and in part because containers do not offer much in the way of security guarantees (precisely because all containers running on the same host share the kernel), whereas VMs do.

Ah, you're using it to mean "no virtualization". Got it, thanks.

I tend to use the term to mean single-tenant, user-supplied OS, non-virtualized myself but I see where you're coming from.

All of that may be true as well - if the user picks the underlying OS (i.e. kernel) to run the container engine and installs on bare metal, then literally everything you wrote applies.

My workstation runs multiple processes in multiple namespaces without containers. Is it bare metal? What if I install applications via flatpak? Still bare metal? Now what if I use docker?

It's tempting to say that containers don't have any virtualization costs, but they often do for networking. For some applications, the addition of another networking driver doesn't matter. For others, it does.

There is no "another networking driver" in containers. It's the same networking driver.

For standard Docker container deployments, there is. See: https://docs.docker.com/network/

That's not a "driver" in the traditional sense, meaning it doesn't talk to the actual hardware directly. It's just a bridge or iptables.

shrug It's additional software in the networking stack, and it has a performance penalty. I think it's fair to call it a "virtualization cost," since you're paying for the container's networking abstraction.

I think GP meant in the sense that Docker containers are just normal programs whose instructions are being executed directly by the CPU (just like how my laptop is executing Firefox as I type this) without relying on hardware virtualization features.

Yeah i was confused by this too. Parent possibly thinking of virtual machines?

Fun fact: all of Google (including Google Cloud) is run in containers. IIRC even VMs are run in containers. The VMs, of course, could contain customer containers, too. It's literally containers "all the way down".