I often think about this. We have 100s of services running on 1000s of servers and 10,000s of containers but I wonder how much of that is ‘waste’ in the sense that we are running 10,000x Operating Systems, 100s of load balancers, database per service/region etc.

Seems possible that horizontal scaling at every layer can be wasteful, but the alternatives are hard for me to even conceive.

Regarding container processes, it depends on how your containers are written. In the best case, each process is not so much an OS as a kernel instance - a kind of poor man's unikernel. It's an extremely stateless process that requires the bare minimum of the physical host.

The overhead that concerns me more is messaging. A network of N nodes has N! paths. In the general case it doesn't take long before the overhead of internal messaging absolutely dominates all other processing in the steady-state. Most people take a brute force approach of partitioning the network on purpose, or even intentionally bottle-necking (again) all the traffic. What's really hilarious is when you see people architecting microservices with kafka and apogee with all the uservice trimmings, only to deploy everything to a single physical rack, or even a single beefy machine.

I feel like the proper path to scalability goes through repeated breakage, because then you get to see how things actually break under load, and what can be done to fix it. For example, you can do a lot by moving logic to stored procs, or equivalently, moving your db into your application process. But people seem to think these are non-starters, for some reason, probably because the notion of stateless app servers as the key to horizontal scalability has become a Law, even though there are alternatives. But good luck questioning foundational assumptions - the risk is just too damn high.