> What yields the lowest risk - spending a ton of time hardening one cluster, or building multiple clusters to reduce the blast radius of bugs and misconfigurations?

Not sure this is a valid dichotomy.

If you are spinning up multiple clusters, you are presumably doing so in an automated fashion. If so, then the effort of hardening is very similar. It doesn't really matter where you do it.

Multiple clusters may have a smaller blast radius, but will have a larger attack surface. Things may be shared between them (accounts? network tunnels? credentials to a shared service?) in which case an intrusion in one puts everyone else at risk.

> If so, then the effort of hardening is very similar. It doesn't really matter where you do it.

Nope. If the clusters are separate it limits how damaging a compromise of the cluster is. This is why cloud providers don’t stick you on the same k8s cluster as another tenant.

> Multiple clusters may have a smaller blast radius, but will have a larger attack surface. Things may be shared between them (accounts? network tunnels? credentials to a shared service?) in which case an intrusion in one puts everyone else at risk.

It’s not really clear what you’re trying to say here. If someone compromises credentials shared between all clusters that’s the same as compromising credentials used by one mega cluster.

> Nope. If the clusters are separate it limits how damaging a compromise of the cluster is.

But if the clusters are configured similarly, a flaw in one is likely present in the others. GPs point is that if you invest in hardening, you can easily apply it to multiple clusters.

> It’s not really clear what you’re trying to say here.

I assume they mean having more clusters present means there are more opportunities to be compromised (e.g. more credentials to leak, more API servers to target, possible version skew, etc.).

> But if the clusters are configured similarly, a flaw in one is likely present in the others.

That doesn’t matter. The point is that you isolate applications/tenants into different clusters. So if someone exploits their own, they haven’t gained access to some other application.

> assume they mean having more clusters present means there are more opportunities to be compromised (e.g. more credentials to leak, more API servers to target, possible version skew, etc.).

That doesn’t even make sense though. In our strawman scenario these are cookie cutter things. Many is not more vulnerable than one in this case.

You can't skip "spending a ton of time hardening one cluster" anyways.

Having multiple clusters may help reduce the blast radius of _certain_ attacks, to some degree. However, managing multiple clusters is a lot more difficult than managing one, and you will potentially replicate bad practices, vulnerabilities to multiple places and increase maintenance burden.

If I could go back, single cluster. Any benefits you get from going multi-cluster can be achieved by configuring a single cluster correctly.

The one benefit you get is protection from bugs in Kubernetes itself and a reduced blast radius. Even if you could produce a secure and H/A cluster, you still leave yourself open to Kubernetes bugs and configuration mistakes such as adding a network policy that blocks all communication across all namespaces.

Multiple clusters protects you from these types of configuration mistakes by reducing the blast radius and providing an additional landing zone to roll out changes over time.

And making it so that "many clusters" look exactly like "one cluster" is one of the goals the kcp prototype was exploring (although still early) because I hear this ALL the time:

1. 1 cluster was awesome

2. Many clusters means I rebuild the world

3. I wish there was a way to get the benefits of one cluster across multiples.

Which I believe is a solvable problem and partially what we've been poking at at https://github.com/kcp-dev/kcp (although it's still so early that I don't want to get hopes up).

If you have 2 clusters, wouldn't you just blue/green them for rolling changes?

Except for security and fault isolation of course.

>a single cluster correctly

Can you elaborate?

At a high level, almost anything you would want to use multiple clusters for can be done on a single cluster, using e.g. node pools, affinity, and taints to ensure that workloads only run on the machines you want them to. As a simple example, you can set up a separate node pool for production, and use node affinity and/or taints to ensure that only production workloads can run there.

One exception, as other have mentioned, is blast radius - with a single cluster, a problem with Kubernetes itself could take down everything.

Another issue is scaling limits. We've found a few dozen ways to break a cluster by scaling along a certain axis. (Most are not related to "vanilla" Kubernetes but the backing cloud provider or specific add-on components.)

Other management tasks are easier when you have separate clusters, such as applying environment-specific OPA policies and not having to filter them based on labels or annotations you hope everyone is using correctly.

At our very large org we do both. At least two clusters per region to isolate platform changes, all hardened to the same standards using automated tooling.

with the usual caveat that it depends on your threat model, I'd say that having separate clusters is likely to provide better segregation.

Of course that reduces the benefits of Kubernetes from a cost perspective, and increases administrative overhead, so it's a trade-off.

Hmm... seems to call for a tool for a cluster-of-clusters.

If "kubernetes" is Greek for the ship's pilot, what's the name for the captain or maybe the admiral of the armada?

"Nauarch, in ancient Greece, an admiral or supreme commander of the navy, used as an official title primarily in Sparta in the late 5th and early 4th centuries bc." - google cites britannica (!)

Cluster-as-a-Service :-)

Hive for OpenShift [0] - Provisioning of the K8s Fleet

Advanced Cluster Management [1] - Management of the K8s Fleet

[0] https://cloud.redhat.com/blog/openshift-hive-cluster-as-a-se...

[1] https://cloud.redhat.com/products/advanced-cluster-managemen...

[Disclosure: I'm an IBMer]

One of the several tools in this space is called Admiralty:

https://caylent.com/kubernetes-cluster-federation-with-admir...

Google says the Greek for that is Ναυαρχείο. Now if you see a product named Navarxeio or Navarcheio, you'll know what it is.

It exists, sort of.

https://cluster-api.sigs.k8s.io/

More directly relevant is Cluster Federation:

https://github.com/kubernetes-sigs/kubefed

Rancher.

This is how GKE Anthos works on prem

Both for different reasons!

what about tooling that hardens it across clusters?