I see some clouds deploying servers with 100 Gbps NICs and I wonder what percentage of deployed applications could get anywhere near that…
I agree, but the number one culprit is premature distribution, thanks to the widely pervasive cargo culting Amazon-style microservices.
Seems possible that horizontal scaling at every layer can be wasteful, but the alternatives are hard for me to even conceive.
The overhead that concerns me more is messaging. A network of N nodes has N! paths. In the general case it doesn't take long before the overhead of internal messaging absolutely dominates all other processing in the steady-state. Most people take a brute force approach of partitioning the network on purpose, or even intentionally bottle-necking (again) all the traffic. What's really hilarious is when you see people architecting microservices with kafka and apogee with all the uservice trimmings, only to deploy everything to a single physical rack, or even a single beefy machine.
I feel like the proper path to scalability goes through repeated breakage, because then you get to see how things actually break under load, and what can be done to fix it. For example, you can do a lot by moving logic to stored procs, or equivalently, moving your db into your application process. But people seem to think these are non-starters, for some reason, probably because the notion of stateless app servers as the key to horizontal scalability has become a Law, even though there are alternatives. But good luck questioning foundational assumptions - the risk is just too damn high.
"What is are the natural, performant operations for the layer below me? How can I construct my solution from these operations instead of pretending the layers are orthogonal?"
A good example would be the linear read/write performance of harddisks. If you had the option to avoid random access and take advantage of read-ahead and other methods you'd have seen much better performance than an approach that ignored this behavior.
There are many, many examples of this.
memory bandwidth bound
memory size bound
cache hit rate and bandwidth bound
TLB size bound
CPU decode and issue logic bound
CPU renaming / OOO buffer bound
kernel implementation bound (esp. locks, interrupts,...)
network physical layer bound
GPU/TPU bound (which are variations of the above, though mostly memory)
power and cooling bound
...
... so basically unless you're running out of functional units, you're hopefully I/O bound depending on where you consider i/o.
But in reality, most SW is just "crappy SW making poor use of resources bound." That's where we mostly are now as an industry. Bad language choices, terrible design, no cross-layer comprehension.