It seems they value developer productivity and innovation over machine performance. I find this enlightening.

I value developer productivity over machine performance too, but they're doing the opposite here: developer productivity is not helped with an unnecessary second layer of containment.

With containers on top of VMs, you now have to manage which containers run on which VMs (and the cloud provider worries about physical boxes), with a pure container solution you just spawn containers (and the cloud provider worries about physical boxes)

I've been developing for a while with a simple Dockerfile in the root of a given project's output... from there I rarely care about it.. in dev, I can mount the app path to the container for dev, and use environment variables set for any other services that are needed. I'm much more productive with Docker for (Windows|Mac) than I've ever been with the likes of vagrant, or whatever bastion of technology that manages VMs... I don't have to think of the VMs, that's generally IT/Infrastructure/DevOps job. yeah, sometimes I wear that hat, but not when I'm coding.

It is helped if the second layer further automates deployment. I would imagine Netflix is better at understanding its velocity than we are.

Yep but Netflix also has a decade of large scale arch wielded to AWS that we don't.

They might not be deploying containers on VMs on bare metal because they want to, but because they /have/ to.

Mind quantifying this overhead you assume containers-on-metal vs VMs will solve?

It's not assumed, it's widely known VMs have poor IO performance, that one of the main points for containers in the first place. Short ver: GIYF.

If you can't provide numbers, then please talk about what technologies you're assuming are involved that create this overhead.

Compute is cheap. People are expensive.

> Compute is cheap.

Not on AWS.

Word on Hacker News is that Netflix is one of AWS' premiere showcase customers and gets discounts that we couldn't dream of.

If you have dynamic workloads, need to be HA, scalable and multi region, AWS is worth every cent.

If you are running a webapp and a database with a known workload, not so much.

The refrain of shitty developers everywhere.

Performance has never mattered. That's why Internet Explorer's map access implementation was exponential instead of logarithmic (to the size of the map).

That one piece of lazy coding held back Javascript on their platform for years, and I'd argue that it as the most significant cause of their browser's market share plummeting (or at last tied with lack of adblockers).

Seems fine for server-side development. For client-side, please don't kill my battery.

Could you elaborate on why containers don't perform on EC2?

I'm not running that combination myself so I wouldn't really know, but I'm not aware of problems with that specific combination or can think of anything obvious.

I don't know about performance specifically, but you lose a lot of flexibility in managing containers when running them on the public cloud. e.g. there are rate limits so a lot of multicast protocols cannot be used as they quickly saturate that and can't be used effectively (IMO).

Containers use LXC in the Linux kernel underneath. Overhead is even lower than HVM virtualization which is a couple percent for most things. Containers are really just a smarter way of dividing resources between users on a shared linux box, something that's been going on since the dawn of time.

It might add another 1% overhead for most tasks to run containers on HVM virtual machines.

The one giant exception is network performance. The network is usually virtualized at the VM level, unless you have an "enhanced networking" VM with SR-IOV enabled. For containers it's virtualized a second time.

This makes the combo potentially terrible if you're trying to run high bandwidth stuff on low performance VM's.

I still like the combo because it allows you to give the big ole finger to AWS if they try to lock you in one day. Since your containers are isolated from the VM you can easily spin them up pretty much anywhere else.

There is a lot of incorrect information in this post.

- Containers are a combination of namespaces, cgroups, and chroot (maybe). You don't need LXC to use containers. Docker doesn't even use LXC.

- There is no overhead for running processes in containers.

- There is no requirement to virtualize networks for containers. They can be configured to use the host's network directly, at which point you are bound by the host's network capabilities. Otherwise it is typically a combination of bridges and overlay networks for which the benefits outweigh the performance concerns for most workloads.

I agree that the typical NAT or SDN setup around container networking could impact performance or at least require additional resources.

But I don't see how that would be any worse on EC2 compared to bare metal or any other hypervisor/provider.

Maybe I'm just interpreting too much into the OPs wording and he did not mean that it's a specific EC2 issue.

I've mentioned this in another comment but the short answer is: rate-limiting.

Now, Netflix, being a priority customer, may get higher limits and such. But average joe public cloud user should keep that in mind before trying to use EC2 for running containers.

Even if that claim wasn't wrong, it's an unrelated question. If there were rate-limiting problems, they'd apply to using EC2 at all even without involving containers.

I think you're ignoring the fundamental issue when deploying container based services v/s services on a multiple VM's. Usually, the architecture for containers involves spinning up a bunch of VM's and deploying some kind of layer on top of that (either K8s or Swarm or something else). When you deploy containers, they may not be on the same VM, or the overlay network itself may require some kind of communication to another container on another VM. This usually creates a lot more communication b/w hosts and rate limiting becomes the bottleneck.

Do you have any evidence of this rate-limiting showing that it's that much of a problem? People have been running clustered apps on EC2 for over a decade and it's not like you hear people saying you can't run Cassandra, ElasticSearch, etc. on EC2 because the network is limited.

Similarly, do you have any data showing that a container system has such incredible overhead compared to the actual application workload? I mean, if that was true you'd think the entire Kubernetes team would be staying up nights figuring out how to reduce overhead.

You run a compute pool, you don't spin up EC2 instances on demand for this kind of application. You scale the pool based on target utilization metrics.

I was wrong about docker, back when I was playing with it it did use LXC, and appears to have started out as project to make a specialized version of LXC. You're right that Docker has its own container runtime now.

The overhead for running containers is usually very low but real. The OS needs to partition low level resources that are normally shared and the scheduling introduces some overhead.

I disagree about network performance. The virtualization adds a somewhat small but non-trivial overhead here (the overhead for other stuff could probably be considered trivial)

Here is a paper I dug up on that gives results to back up my ranting. It's a bit old now but probably still holds mostly true. http://domino.research.ibm.com/library/cyberdig.nsf/papers/0...

I'd need a citation that a process running in a namespace adds overhead.

My point about network virtualization is that it is not required to use linux containers. Yes, some container tools do create network abstractions that add overhead, but they aren't required and most tools allow you to optionally bypass the abstraction and sit directly on the host's network stack.

AFAIK the early go versions of docker used LXC underneath, but that was ~3+ years ago

Sorry . It's hard to keep up these days, I'm 112 in Javascript years.

Wow. Your knowledge of computers is excellent for only being 4 months old.

Containers are simply processes so NOT 1% overhead! No overhead but get hard to believe.

Late edit: I've said 'Red Shift' above, I mean Red Hat's OpenShift.

Tech has too many edgy code names.

.. so kubernetes.

Kubernetes on bare metal, with additional tech to isolate VMs from each other safely. See link elsewhere in thread.

SELinux is NOT isolation. The main issue is the large kernel attack surface and SELinux, while important, only solves a small part of that.

That's true, but a lot of the classic containers-not-containing issues (/sysfs hacks to get into the parent kernel etc) are prevented by SELinux policies.

See https://blog.openshift.com/securing-dockers-future-with-seli...

Agreed.

But people keep selling SELinux or AppArmor as a solution for multi-tenant container environments, which is just plain false.

The real solution are efforts like like Intel's Clear Containers and Hyper's runV.

We get fast performance when we need to, and there's a number of EC2 technologies that help with that.

Or use something like GKE where the containers should be running directly on hardware, so you only have one layer.

"Bare-metal" means running directly on hardware. But yes, GKE would also join Triton and Red Shift (provided it has similar container isolation capabilities).

Actually, as 0x74696d mentions below, GKE is also not bare metal.

Doing a little research: GKE runs on https://cloud.google.com/container-optimized-os/docs/ which is designed for GCE, which runs on KVM: https://cloud.google.com/compute/docs/faq

So it looks like just Triton and Red Shift.

Disclaimer: I used to work at Red Hat, which means I like the people behind Red Shift, but that also means I hate the people behind Triton as part of the early 2000s Linux/Solaris wars.

GKE clusters run on VMs, not on bare metal.

Containers don't actually run directly on hardware with GCE - there's still a virtualization layer in-between.

I'm 99% sure that Google runs one VM per container because that's the only way to make it safe.

Anything else would be insane.

> I'm 99% sure that Google runs one VM per container

I'm 100% sure you are wrong. You might as well just use VMs.

Containers are not only about safety, you know.

I stand corrected, it's a bunch of VMs per customer but still no multi-tenancy.

1 container per VM? What's the point? If you can bypass container sandbox it's very likely that you can do the same with VM.

> If you can bypass container sandbox it's very likely that you can do the same with VM.

Hypervisors are much, much harder to break out of than a Linux container.