I don't know what your internal architecture looks like, but on the face of it, arguably you should be running Dev and QA loads on the same cluster. Either you have an organization where an SRE team is responsible for running clusters for teams, in which case why not run Dev and QA on the same cluster, or you are responsible for last-line-of-support for teams responsible for running their own clusters, in which case you say, here's a new version of Kubernetes (e.g. a Terraform module) and in this sprint you are responsible for deploying it through dev, QA, pre-prod, and production clusters. Especially if you have separate Kubernetes clusters per product line (do you really need separate dev Kubernetes clusters per product line?).

unless you have some strict security reasons, why even have separate prod clusters per product? We definitely don’t. seems to kinda miss the point of k8s - aka running all kinds of mixed workloads with separation within a cluster if you need it.

Agreed. Separate prod clusters per product usually fixes an organizational problem (lack of trust that running shared workloads would/could be safe, versus giving each team their own servers) first. A lot of organizations, sadly, prefer to pay for separate clusters than to set up RBAC, ResourceQuotas, etc.

Hypothetically, since the scaling limit for Kubernetes clusters is ~10,000 nodes (last I checked), you could have multiple product lines that took up 10k+ nodes each. Then there's no reason why not to split by product line. But in the beginning it should be fine.

There's also edge cases - Kubernetes doesn't support setting resource requests or limits for networking or I/O, which you usually solve by setting up taints/tolerations/affinity to manually schedule those workloads onto nodes where you've manually run the numbers. But still not usually a reason to prefer separate production clusters.

Arguably, the design of kubernetes is for clusters to be based on physical clusters - to the point of a cluster per aisle (or multiple aisles), and resources of those clusters then being used to deploy applications even across clusters. Or in the small, like Chick-Fil-A, with kubernetes running locally at every restaurant.

Yup, this is how you scale an application past the 10k node soft limit.

Not just past 10k node limit - it's also because the k8s design has unsaid assumption about being used to arrange physical sets of machines, similar to how Borg paper would talk about cluster being equivalent to an aisle in datacenter.

Are you just referring to how you assign labels and taints to nodes, which in a physical datacenter could include labels like which aisle the baremetal server is in?

Of course that's feasible, to tell the Kubernetes scheduler that you only want to schedule the workload on a very specific server, but that's not really best-practice...

No, I'm referring to bundling machines into clusters across physical lines, with bare metal deployments.

Easier to track expenses (for cost reporting etc) : each products are billed separately

Easier to manage because less dependencies : you are the sole user, so you can do whatever you want

This is the greatness of public cloud, where multi team synchronisation can be reduced a lot

For very small projects, one cluster per product is expensive, so mutualisation makes sense. Past a certain size, this is not the case

> separate billing

IME mixing financial concerns with engineering concerns puts you into an architectural corner. Suddenly the business wants an integration between the two projects - which cluster does the integration run on? Do you spin up a whole new cluster / account just for the integration? Inside a project, you need to run dedicated infrastructure for large customers. How do you report that for billing?

Just set up tagging correctly.

> For very small projects, one cluster per product is expensive, so mutualisation makes sense.

If you're not working for a BigCo, every project starts out very small ;) You shouldn't optimize prematurely.

I meant "small" as in "can live on a 3*4GB RAM cluster"

About integration, I do not understand what you say You have a provider and a consumer, each of them build (and pay) their own parts

I've worked on ETL integrations in the past where there's an API for service A that's read-only and an API for service B that's write-only, service A is provided by a third-party vendor (so you're not exactly going to get the vendor's code to alter or fork it) and politically the people who work on the integration are different from the people who are responsible for service B; the integrators neither have permission to make alterations to service B nor will they get service B engineers to take backlog items to integrate read functionality or otherwise integrate with the vendor for service A directly. The integrators simply need to build a completely separate integration that lives outside of service A and service B to ETL from service A into service B.

If service A is in one cluster, and service B is in another cluster, where billing is per-cluster, in which cluster do you put the ETL integration? It belongs to either both or neither, depending on how Finance wants to bill it.

yeah, billing is pretty simple with a bit of investment on tagging. If it gets complicated there are third party tools to help. Much better than introducing significant complexity into your systems architecture just for the accounting team.

Sometimes certain use cases require clusters running in different regions of the world, or have to run in a specific region. There's also fail over. Personally I always strive to use a "mono" cluster unless I can't avoid otherwise. It does happen.

we have hundreds of clients most on k8s or openshift. Many of them are shifting to smaller team or division oriented clusters so that they can move at their own pace in consuming capabilities, tools, security practises, etc. It also simplifies distribution of operational expenses, yea there’s tools out there to help with that but it’s a whole lot easier to hand someone a sub-account in AWS and call it a day. Additionally it reduces your fault boundary. It isn’t as sexy as saying you have a 5000 node cluster with 100k+ pods running but it can make life a whole lot less stressful when it comes to upgrades and changes.

imo you've just traded one set of centralized costs (developing tools, security practices that work for all and educating people) for a distributed set of costs with everyone managing their own clusters and duplicating all this work. You've just pushed the cost onto other teams - which can be a fine approach for reducing the central teams stress, but is net more costly to the business. EDIT: Im not saying its necessarily a bad choice, just much more expensive.

It’s all swings and roundabouts has been since the broad adoption of computers. I expect in a few years time when everyone’s forgotten the pain points that drove them to decentralize there’ll be a big move to centralize again. Someone will get a gold star for coming up with the idea and being able to demonstrate the “cost savings”. Of course it’ll completely ignore the general disruption to all of the product teams as they adapt to the new world order but what’s a company without a little busy work.

So basically, doing to infrastructure the microservices approach?

a) if you're going to intentionally make the trade-off to have higher costs in exchange for reducing ops load and simplifying administration within an account, where services make network calls across accounts, then adopting ECS/Fargate is probably a significant improvement over Kubernetes.

b) The underlying engineering / financial reality is still there and very much a leaky abstraction that will show up on your AWS bill. Cross-VPC, cross-region, and cross-AZ networking costs are very real overhead that you must consider; you either still have centralized network planning with shared VPCs and subnets or you basically decide to give up and let AWS send you the bill when teams create their own VPCs, their own subnets, etc.

c) setting up DNS / service discovery and network controls within a single cluster is simple. Doing so across account boundaries is not, and deciding to set up solutions like AWS Transit Gateway incur their own costs.

I'm sure it simplifies some things for some people, but Finance is going to get upset pretty quickly. Once you start to go down that path, it's very difficult to back out.

Everything’s benefits and tradeoffs. With this model there’s a clear ownership model. I’m not suggesting it’s right for everyone but it is a trend we’ve noticed with a number of clients over the last 2-3 years.

For sure, with scale comes slow-downs and new problems, that’s just an organisational reality, totally get that.

I am slightly curious why an upgrade takes 2 weeks? At my current work, rolling out an upgrade (self managed clusters), rolling out an upgrade on the “happy path” is a fairly low-intensity task: we kick it off, nodes gracefully drain, upgrade and come back online. No intervention necessary. Unhappy path just requires another command to roll them back. Prod is the same, but happens slightly slower because there’s more nodes (and more independent prod clusters).

Why don’t you update dev+qa at the same time? Or qa+sandbox? Or dev+qa+sandbox? Separating these 3 might make sense for the applicative part (and even then, I think 4 environments is too many), but I’m not sure it really make sense to follow it on the infra side.

I’m also not sure of why you manage all these clusters, why not merging dev, qa and sandbox in a single cluster with namespace partitioning? It would be way less work and probably more cost effective