This is what the CRI does/is, basically. Various projects sprung up to make it possible @ the runtime/kubelet level (kata-containers, frakti, containerd untrusted workloads), but support for runtimeClass[0] is what's going to tie it all together and it's already in Alpha.
Personally I think we should be moving away from docker files -- docker's superior ergonomics pushed the industry forward at the outset, but they lagged in features and compliance with any standards for a long time and were basically usurped. The Dockerfile is a decent format but it lacks a lot of good qualities, like being trivially machine-editable, and the docker client itself has some unfavorable tradeoffs when compared with tools like rkt and podman. I think it's a mistake to standardize on Dockerfiles, but it's definitely a good idea to standardize on the CRI (which Docker now adheres to via a containerd shim by default).
The best thing by far about kubernetes is the CRI[1], CNI[2] (Container Networking Interface), and CSI[3] (Container Storage Interface) standards that are coming out of it. I don't think any one realizes it yet, but they are inadvertently building secure/sandboxed computing for everyone. Containers are just sandboxed processes, and before this, most people were running basically completely unsandboxed processes (both in the cloud and on their personal computers) -- once all this stuff lands, linux is going to have some amazing features for running applications more safely -- production-grade safety for any application you run on your own machine available with standardized tooling.
[0]: https://github.com/kubernetes/enhancements/blob/master/keps/...
[1]: https://github.com/kubernetes/community/blob/master/contribu...
Assuming you're deploying your Kube cluster in the cloud, the costs of having multiple clusters is really reduced. You don't have to allocate physical machines or worry about utilisation as much - you just pick a node size and autoscale.
What that enables is thinking about other concerns when deciding how many clusters and where they are is right for your team.
There are operational reasons why having multiple clusters is a good idea. At the simplest level, making a config change and only risking a portion of the infrastructure is an example.
As an aside, something that's useful when thinking about Kubernetes multi-tenancy is to understand the distinction between "control plane" multi-tenancy and "data plane" multi-tenancy. Data plane multi-tenancy is about making it safe to share a node (or network) among multiple untrusting users and/or workloads. Examples of existing features for data plane multi-tenancy are gVisor/Kata, PodSecurityPolicy, and NetworkPolicy. Control plane multi-tenancy is about making it safe to share the cluster control plane among multiple untrusting users and/or workloads. Examples of existing features for control plane multi-tenancy are RBAC, ResourceQuota (particularly quota on number of objects; quota on things like cpu and memory are arguably data plane), and the EventRateLimit admission controller.
There's active work in the Kubernetes community in both of these areas; if you'd like to participate (or lurk), please join the kubernetes-wg-multi-tenancy mailing list: http://groups.google.com/forum/#!forum/kubernetes-wg-multite...
Also, I gave a talk at KubeCon EU earlier this year that gives a rough overview of Kubernetes multi-tenancy, that might be of interest to some folks: https://kccnceu18.sched.com/event/Dqvb?iframe=no (links to the slides and YouTube video are near the bottom of the page)
Disclosure: I work at Google on Kubernetes and GKE.
Many teams use clusters for stages because they work on underlying cluster components and need to ensure they work together and upgrade processes work (e.g. terraform configs comes to mind). Theres no reason to separate accounts because the cluster constructs aren't there for security.
Considering it deeper (I haven't had to think about this for a while), I think multi tenancy would cover almost all of the use cases I've seen except for the platform dev where people use clusters for separation when testing cluster config-as-code changes.
The idea being that you have process around getting your code to run on the livedata cluster and this we add more stringent requirements for accessing each API.
This is for soft tenancy, and you want to write admission controllers to reject apps that haven't went through the defined process.
Edit: looking more in the thread, you clearly know this much better than I do. I'd like to get the chance to talk and improve my understanding, if you ever find some spare time.
As a gut check, most of these reasons apply to AWS AZs as well. If your Kubernetes strategy calls for more than one per region per AWS account, it means that you, organizationally, don't trust containers as much as you trust VMs on AWS. And right now, you're probably right to do so.
Treating a single cloud account and the clusters it hosts as a single security cell is more than reasonable, especially if you need to handle sensitive data, be able to audit the state of the cluster (hope your cluster can’t overwrite the contents of the S3 bucket that you’re using for cloudtrail logs, etc), or reason about the risk of compromise.
We run lots of multitenant kube clusters, and I would still recommend anyone who expects to grow to create hard walls between clusters and cloud infra as soon as reasonably possible.
Multiple Kubernetes clusters for workload separation and AWS accounts for workload separation aren't quiet the same and bring different levels of complexity depending on what your internal processes look like.
I am a firm believer of Borg style cluster OS, what Borg lacks is a modern API, but the fundamentals are already there.
Disclaimer: I am with Google's Borg team, with a focus on its client side.
It is correct that containers leak, and people know this. Multi-cluster strategies are real, and they shouldn't be. It should be OK to have one big cluster[1]. Until Kubernetes fixes this, there will be some friction to adopt it, based on real use cases like untrusted code and noisy neighbors.
It is incorrect because users (e.g. non-infrastructure engineers) don't know or care about the precise definition of containers and VMs are. The point of "containers" is that I can define something that acts like an operating system from the ground up, and it builds quickly and runs quickly in production.
Kubernetes doesn't win by forcing users to think about VMs. Kubernetes wins by adopting a VM standard that can be built by Dockerfiles. Infra engineers will love it.
But besides them? Nobody will care, because Docker for Mac will look the same.
[1] Maybe 1 cluster per region? There's a whole fascinating topic that starts with the question "when building a PaaS, do you expose region placement to devs?" The answer implies a ton of stuff about what exactly it's reasonable to expect from a PaaS and how much infrastructure your average dev has to know.