Other way around: Everything at Google runs inside a container, including the VMs

gVisor lets you run multiple untrusted workloads on the same VM, in this case a GKE node.

What would running a VM inside a container provide in terms of security and isolation that just running a VM would not?

This ACM article from a few years ago written by folks that worked on Borg/Omega/Kubernetes states:

>"The isolation is not perfect, though: containers cannot prevent interference in resources that the operating-system kernel doesn't manage, such as level 3 processor caches and memory bandwidth, and containers need to be supported by an additional security layer (such as virtual machines) to protect against the kinds of malicious actors found in the cloud."[1]

Also see slide 13 of Joe Beda's talk from a five years ago shows the container running in a VM not the other way around:

https://speakerdeck.com/jbeda/containers-at-scale?slide=13

[1] https://queue.acm.org/detail.cfm?id=2898444

(I work for GCP)

It looks something like this:

your container -> Compute Engine VM (GKE Node) -> container -> Borg

The container on top of Borg is used for scheduling and management. Joe's talk has a slide on this. As a GCP customer, you never have to worry about this or care about it, as it is an implementation detail.

>"The isolation is not perfect, though: containers cannot prevent interference in resources that the operating-system kernel doesn't manage, such as level 3 processor caches and memory bandwidth, and containers need to be supported by an additional security layer (such as virtual machines) to protect against the kinds of malicious actors found in the cloud."

As a GCP customer using GKE, your applications are separated from other GCP customer using VMs.

However, if you want to run your OWN untrusted workloads, then in the past you would have to spin up a separate VM for untrusted workload A and a one VM for untrusted workload B.

This sucks in terms of resource utilization. It would be better in many cases if you could run workload A and B on the same VM. That's where gVisor comes into play.

your untrusted container -> gVisor -> Compute Engine VM (GKE Node) -> container -> Borg

I hope this makes sense!

Thanks for the explanation, this makes sense yes.

>"The container on top of Borg is used for scheduling and management."

Is this the "open source node container manager" box on slide 13 then? I'm guessing this is the Borg's version of the kubelet then?

https://speakerdeck.com/jbeda/containers-at-scale?slide=13

That's a very old slide :) I "guess" the slide deck was talking about https://cloud.google.com/compute/docs/containers/deploying-c...

I see. So is "the container on top of Borg is used for scheduling and management" the Borg equivalent of the K8S kubelet then?

As you can see from the Borg paper [1] and the name, "borglet" is the most closest component to "kubelet".

[1] https://pdos.csail.mit.edu/6.824/papers/borg.pdf

(Co-author of the post)

The fact that gVisor is being used in multiple services at Google is probably the confusing part. In case of GKE Sandbox, the users here are external and using Cloud (specifically GKE). The target use case is to add defense in depth to their pods running on potentially shared GKE Nodes (VMs) for Multi-Tenancy. Our talk at Next'19 [1] includes a story by one of our customers, which may help understanding the use cases.

[1] https://www.youtube.com/watch?v=TQfc8OlB2sg

Thanks for the link that does make the use case clear i.e multitenancy/SaaS. Am I correct in assuming though that when someone creates a K8S cluster via GKE that the containers that make up their cluster such as the kubelets and masters are all running in VM underneath?

Yes, exactly. https://cloud.google.com/kubernetes-engine/docs/concepts/clu...