I run a small cluster for my side projects consisting of 3 nodes. 1 node is dedicated to run database containers and control plane node (it's also a worker node but strictly for database statefulsets only), and the other 2 as worker nodes, but one of them also run an NFS server that mounted as persistent volumes so every worker nodes can have access to it.
Self-healing? Internet-scale? I have on interest for them on my small cluster. I just want my apps to not go down while I'm updating them without writing complicated blue-green deployment scripts, ability to move around pods to other nodes when one got overloaded, and ability to add or remove nodes when needed without starting again from scratch. I basically treat it like docker-compose on steroid. So far it works really well.
It's "Amazon ECS", and it works pretty well for standard fare CRUD web services, but more complex use cases quickly end up pulling in more and more AWS services (for example, if you need to run a cron job, you can use EventBridge to trigger an ECS task or just do it all with Lambda). This isn't dramatically worse--it's mostly just different. Kubernetes APIs are more consistent than in AWS, and Kubernetes tends to be more flexible and less limited than AWS. It's also much easier to extend Kubernetes or find a pre-existing extension than in AWS. But mostly it's not going to make or break your project one way or the other.
If you're just running CRUD web services, this is fine, but if you need to spin up a background job or do more complex orchestration then it can quickly become advantageous to
But what would you suggest in lieu of Kubernetes? What would save work for a shop which is not yet a giant but has already overgrown the capabilities of 2-3 manually managed boxes / instances?
I can think of several options. Management by Ansible / Saltstack / Chef can easily become a rabbit hole comparable to maintaining K8s's YAML. In your experience, does Nomad save SRE work at smaller scales? Does Terraform? CloudFormation?