I don't think this is really true, but I'm open to hearing your thoughts on this. There was a bug in their HTML parser which caused unrelated memory to be dumped to the process. Their SSL termination servers were isolated elsewhere which is why SSL keys weren't dumped into public caches.

Where would you like them to draw the isolation boundary? Per function? Per rule? Per service? From what I understand, these processes were a part of a single service, but not every request was using each type of rule.

Even if they'd only had customers using their html parser isolated on separate servers, other customers would have been affected even if their HTML was perfectly valid according to the parser.

That's roughly what we do, though we run an hosted version of an open source webapp, not a CDN. It's more expensive resource-wise (particularly RAM), but it has meant that we were immune to 90%+ of the security bugs discovered in the platform.

Why would the costs be outlandish? We offer that and we're fairly cheap. Since the cost is mostly fixed per customer, it should scale linearly.

As for scaling, they already have to do that, by pointing different requests at different servers depending on their load, etc.

If there are 1M customers that's a minimum of 1M servers. Some customers are obviously larger and would need more. There's also HA. Let's conservatively call it 2.5M servers.

At an absolute bare minimum we'd need to allocate 2.5M GB of ram and 2.5M vCPU. That's a huge amount of resources.

If you could reliably fit 10000 Small customers on a single server at 32gb ram and 8 cpus you can already start to see how many resources can be saved.

Without customer isolation you've got the entire cluster to handle load spikes and HA. With isolation you have to have scheduling monitoring each of the 1M clusters and scaling appropriately by anticipating demand.

Scaling a service is way easier than scaling customers within a service or many services.

Why? There's nothing magical about a container, it's literally just a cgroup of Linux processes. You don't have to use them to get the memory isolation we're talking about - uncontained processes get it too.

That's what we do: one process per client, uncontained, just running on a different system user.

But in any case, sure, use containers, I'm certainly not opposed to them.

Perhaps for some tier of paid customer.

They certainly have the resources to solve this if only they want.

All I can say is that I am not willing to pay for something so fragile, but that is only my own opinion.

In the real world things may go differently: they exposed critical data back in 2012 too and they're still here ...

Each _process_ has its own memory page table. Containers are built out of processes, so they inherit this attribute.

Namespaces have nothing to do with it.

That just punts the vulnerable code elsewhere. A kernel bug could leak memory across processes. And the kernel is also written in C, so you aren't getting protection from a "better" language either.

I disagree. I think it's much less likely, because the kernel doesn't usually get involved in the process' memory once it is allocated.

Sure, it maps the virtual memory about to physical memory as needed, but bugs there is likely to cause severe corruption resulting in an immediate crash. The kernel doesn't go low level enough for it to be likely to result in messing with a process such that the single process continues to work but also leaks across the process-internal customer boundary that the kernel cannot see. That would require a level of surgical precision I don't think is likely in a bug.

To be clear, I'm not saying that you shouldn't use per-customer processes. The kernel has more eyes and is less likely to be vulnerable in this way. Just that from an analytical perspective, you are really just moving the problem elsewhere, rather than solving it.

Suppose you're a "bad actor", knowing this is a shared service, wouldn't you look for 0 days in it ? A carefully crafted exploit has the potential to leak specific content from unaware customers again.

The attack surface is nginx (http://nginx.org/en/security_advisories.html) plus each component of each loaded module ...

It would be saner to apply isolation to each element of the cartesian product between customers and services.

The performance (and cost) impact can be mitigated by scheduling resources over a pool of disposable virtual machines (obviously in xen and with iommu protection), but I bet they can develop even better solutions.

The operational overhead of VM/Container isolation for the cartesian product of customer + service sounds like it'd be extremely prohibitive. It's certainly a tradeoff, but to claim it's saner is missing all of the other costs associated with such a system.

Maintaining a running pool of VMs per service in a sufficient number to serve the load of requests grouped by customer, and assigning the VM to a specific customer only at needs is different than running permanently a pool of (n).customers x (m).services VMs.

This is why an efficient scheduler and the usage of disposable VMs is a need. Still depending on the load and the variety of the traffic it may not be feasible, you are absolutely right !

Another approach to ensure isolation is the usage of a MAC framework. As I wrote "I bet they can develop even better solutions" ;)