It should be also known that all "run your code on machines" platforms (like ECS) have similar issues. I remember using ECS pre-fargate and dealing with a lot of hardware issues with the instance types we were on. It was a huge time sink.
> it's only quick if you deploy weave-net and call it a day
That's exactly the benifit of kube. If something is a pain you can walk up to one of the big players and get an off-the-shelf solution that works and spend very little time integrating it into your deployment. No cloudformation stacks or other mess. Just send me some yaml and tell me some annotations to set on my existing deployments.
> I would strongly discourage anyone using k8s in production unless it's hosted or you have a full team whose only responsibility is it's maintenance, provisioning and configuration
If you have compute requirements at the scale where it makes sense for you to manage bare metal it should be pretty easy for you to find budget for 2 to 5 people to manage your fleet across all regions.
Plus disrupting all the developers.
So far every large scale implementation i have seen has cost the developers a year in productivity.
It’s not F500 scale, but it’s over 100 CPU scale. Confident I have a ton of room to scale this.
1. Off the shelf software (even from major vendors [0])
2. Hireable skill set: You can now find engineers who have used kubernetes. You can't find people who've already used your custom shell scripts.
3. Best practices for free: zero-downtime deploys can now be a real thing for you.
4. Build controllers/operators for your business concepts: rather than manually manage things make your software declaratively configured.
[0] - https://cloud.netapp.com/blog/cvo-blg-kubernetes-storage-an-...
Doing maintenance on the cluster isn't. Debugging routing issues with it isn't either, configuring a production worthy routing to begin with isn't easy either. it's only quick if you deploy weave-net and call it a day.
I would strongly discourage anyone using k8s in production unless it's hosted or you have a full team whose only responsibility is it's maintenance, provisioning and configuration