Right; and I don't feel that's a knock against K8s; its just the trade-offs it decided to make. A true Google-scale solution would be far worse to use, you can be sure of that.

K8s is a "middle 80%" scale solution. Its not made to run Google (though Google uses it a ton internally). Its also not made for your average four person startup (though, if you've got that experience internally, its not a bad choice; its not Heroku, but its better than a lot of deployment options out there).

All I'd say is: I've worked in a "scale-up" B2B org under $1M in ARR. We were pretty monolithic; just a backend NodeJS app, and a frontend SSR React app, basic. Five engineers by the time I left. We used K8s (EKS+Fargate). Maybe 50 pods total, across two environments. It was fantastic. We never had to say No to any weird customer, product, or engineering decision which would be difficult in either more managed, or more legacy, systems. Customer wants a custom domain and'll pay $50k for it? Like five lines of YAML and update Route 53, done. Datadog sidecar container so we can ingest some APM traces? Ten lines of YAML copy-pasted from their docs, done. Update the cluster? Click one button in the AWS UI. Every developer wants their own staging environment? Ok, bit more work, but: create some namespaces, retool the CI a bit, we can deploy separate databases in there as well its only staging, actually pretty straightforward.

Half the stuff we did with k8s would have taken three times as long with more native AWS solutions, and some of it probably would have been impossible on something like Heroku. K8s strikes a balance. Its not the simplest thing in the world. I wouldn't grab it on day 1 of a startup's engineering journey. But I wouldn't knock a startup who does.

I'm using k8s even for my personal 2 node cluster. It's just so convenient to be able to use all the automation tools.

I can leave the cluster alone for weeks, and it'll take care of itself, my CI will build new docker containers, tooling will start rolling them out across the cluster, if deployments fail they get rolled back, and I get an email, etc.

At some point I was hands-off with the cluster for 6 months and everything kept itself up to date and running just fine.

Google's Borg clusters don't have wildly larger sizes than that despite having more years of development and a lot of motivation at that scale. They instead have a lot of clusters and transparent inter-cluster network topologies (i.e. you can choose clusters with very high bandwidths to each other).

The fundamental design just isn't infinitely scalable, and at a certain point, you might rather have some bulkheads/autonomy or regional diversity.

Curious how you ran 36k node cluster. Did it involve elastic scaling, etc? What are the alternatives for k8s?

Home grown: https://www.semanticscholar.org/paper/Robust-large-scale-ren...

off the shelf: https://hradec.com/ebooks/CGI/RMS_1.0/rfm/User_Interface/Alf...

although that was with something like 6-10k nodes because there was an upper limit to how many dispatches alfred could do because it was single threaded, from the early 90s and not really designed to scale that high

https://renderman.pixar.com/tractor is probably what they use now, or https://www.opencue.io/

but any grid engine style dispatcher/manager will do what you want. It'll give you the primitives to manage wildly larger scale than k8s.

These clusters were on real steel, as elastic clusters were horrendously expensive, and the storage was/is nowhere near fast enough.

Nowadays, I'd use AWS batch, or at a push airflow.

Yeah, I don't know if it's because ECS was my first container orchestration experience but every time I look at teams trying to do k8s on AWS I think how much easier ECS would be.

The complexity difference between bog-standard ECS+Fargate and EKS+Fargate deployments rounds down pretty small. Biggest I've seen: ALB integration, IAM integration, and maybe certificate management. Most of that stuff is out-of-box on ECS, but on EKS you need some extra containers or configuration to watch the K8s API and provision stuff for you (if you want to use it; you can also just go pure-k8s) (edit: just to be clear, they provide all this for you; its not out-of-box, but its easy-to-add-box e.g. [1]).

An argument could be made for something like CodeDeploy being better integrated on ECS, but that's more of a "k8s doesn't need CodeDeploy but ECS might" kind of thing. And even then, I wouldn't touch it.

An argument could also be made that upgrading ECS clusters is a bit easier, as the cluster itself, uh, doesn't have a "version". But on Fargate, its pretty painless on EKS, and Fargate ECS tasks do have a "platform version" that generally doesn't have to be worried about (version: LATEST), but is nonzero nonetheless.

Which is really to say that both ECS and EKS puke complexity, because its AWS, but the volume is pretty similar.

[1] https://docs.aws.amazon.com/eks/latest/userguide/aws-load-ba...

Agreed. I had the displeasure of doing CodeDeploy to ECS Fargate a few weeks ago for a side project and IMO it was overly complex.

Yup. You can get pretty dang far with just "the task definition is in cloudformation, so update the image in cloudformation and submit the template". For a bit more complexity, add ALB routing weights in, and the case for CodeDeploy is kind of weak. My statement that ECS needs it was probably overzealous; for most situations it doesn't, and if there's some crazy functionality it does I'm not aware of, you'd probably need something else in k8s as well (e.g. istio).