> There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies. The first method is far more difficult.

- C. A. R. Hoare

Ex Amazon SDE here. I would pick the first method a hundred times.

People would be surprised at how simple the internal infra is, given the fleet size, compared to stuff like k8s.

(I'm talking about the infra that runs on bare metal, not AWS)

Amazon SDE here. I'm not sure what internal infra you're talking about. Amazon's certainly isn't one of them.

And AWS, with its design by accretion, makes that an impossibility. Kubernetes by comparison is a paragon of clarity.

Why does AWS insist on building this way? Has anyone tried to get the teams together and say, "stop, let's evaluate and simplify where we can?"

As an AWS customer, I find AWS a huge pile of overcomplicated offerings that is dense with its own jargon and way of doing things. IAM is a trash fire. Everything built on IAM like IRSA is also a trash fire. Why are there managed worker nodes, spot managed worker nodes, Fargate worker nodes, self-managed worker nodes, and spot self-managed worker nodes? Why are there circular dependencies for every piece of infra I stand up making it almost impossible to delete any resources? Why can't I click on a service and see how much it costs me in one or two clicks?

This is a completely insane way of building software to me and I have to eat it.

> "let's evaluate and simplify where we can?"

Amazon has "Invent and Simplify" as one of the leadership principles. However, each team has its own hiring bar and culture, so some will take it seriously and some will not.

> I find AWS a huge pile of overcomplicated offerings

I fully agree. I'm speaking exclusively about the internal infrastructure, not about AWS.

As if they are trying to tie them to their proprietary platform to make migration to other cloud providers hard.

I can't even imagine, sometimes I feel like AWS offers this pretty facade of "scale it up easy" and in reality its like "wait, I can't even see why this thing is failing because its a layer below, and AWS support costs mega bucks"

You mean the infra behind non-AWS services like retail Amazon.com?

Or the infra underlying AWS itself?

The first one, but I'm trying not to divulge too many details on the architecture of Amazon - as you can imagine.

Fascinating because I would have thought the whole idea is AWS is dog-fooded by retail. Yet in truth AWS is itself too expensive and / or complex for Amazon itself! (Or at least the original part of it.)

The majority of retail is dog-fooding AWS, but a lot of code was written before AWS existed or before it provided many of the required services.

CAPI is for creating clusters only. It doesn't offer any orchestration.

I'm curious what you mean here, since CAPI allows you to manage and scale clusters like any other Kubernetes object. You can literally "kubectl scale --replicas" a cluster control-plane and worker nodes.

You end up needing this anyway, because Kubernetes clusters are confined to single geographic regions and you need some way of managing the separate cluster(s) you have in each region

Not saying you should, but it's entirely possible to have nodes all over the world managed by a single set of master nodes. The system is not really designed with that use case in mind though, the added latencies will hurt a lot in getting etcd consensus.

While it is certainly possible to have mixed region clusters, from my experience it is better to have one cluster per region. Do your synchronous communication within the cluster / within the datacenter with sub millisecond latencies and do asynchronous communication across cluster boundaries with up to triple digit millisecond latencies.

A Kubernetes for Kubernetes! But what about when we want to upgrade that?

I mean thats simply Kubernetes for Kubernetes for Kubernetes. Lets go!

I need a Kubernetes Kubernetes Kubernetes Kubernetes Kubernetes emulator so I can run Civilization 14 on my iPhone 35XSXS.

Inception!

In a way this reminds me of Kubernetes Virtual Clusters. Each virtual cluster has its own tenant control plane, namespaces. Multiple virtual clusters exist in a super cluster.

https://github.com/kubernetes-sigs/multi-tenancy/tree/master...

I think at a certain point up in the stack you don't want to have another orchestrator for the orchestrator. You would just tell each team to keep their cluster up-to-date without creating a (dangerous) master button to update them all at once.

Wake me up when I can finally run a bare-metal Kubernetes cluster in Kubernetes to get rid of all my hardware.

To ugrade it you summon Scrum of Scrums, the Master Scrum Master has all the answers you might need.

Isn't that basically what helmfile[0] does?

It lets you declaratively define helm deployments, which in turn orchestrate k8s infrastructure.

0 - https://github.com/roboll/helmfile

Agreed, this is absolute insanity and people can't see it from the weeds.

I think we’re already there? Maybe we’re better off looking for a cluster orchestrator orchestrator.

it's kubernetes all the way down

That's what Federation in Kubernetes was for, sort of, but it has never really come to fruition. The closest I've seen is Admiralty, but I'm not sure how it actually works out in reality.

Federation v2 is nice, but not many places use it.

Unfortunately, all my clusters are either confined locally anyway, or are deliberately supposed to be separated in all ways :(

Kubermeta

It used to be called Ubernetes, shame that name didn't stick.

I operate one at work, actually. It's not so bad.

Kubernetes solves a problem. It isn't appropriate for all use cases, but the real problem isn't kubernetes but rather the knowledge gap around new technology.

KubeFed - https://github.com/kubernetes-sigs/kubefed

Rancher does this nicely.