Aren't you contradicting yourself?

If you're on 100M users you're probably scaling vertically. So adding 5x more hardware shouldn't be a problem.

But when you're at 500M all of a sudden it makes sense to optimize further since the capital saved will be the same percentage(ish) but the money is worth peoples time all of a sudden.

I know that we don't care particularly about power savings in the DCs I've worked in, because they're relatively small. While bigtech will do all kinds of shenanigans to save a couple watts here and there, because it's worth it across your hundreds of thousands of servers.

In the real world, it's rare to find a non trivial business where operational costs to its problem size scales linearly. Let's take an example of search. If t he web size grows by 10x; this not only means # of your user grows by 10x but also # of web pages to be indexed, size of the ranking model, ratio of unstructured/dynamic content formats, faster pace of content updates, reduced informational density of overall web, size of user logs, advanced SEO, (those are just a tip of iceberg, there are so many other stuffs) etc etc... Every single dimension scales and stacks up. And you need to handle all of that within a limited latency budget where you cannot really make any compromises.

> So adding 5x more hardware shouldn't be a problem.

If only things were that simple. https://en.wikipedia.org/wiki/Amdahl%27s_law

Seeing scale issues as purely hardware bound is incredibly naive. Even in a case like streaming, if you’re pushing more bits through the wire, it’s likely the increase in usage causing the traffic increase affects the software systems you have in place to support your service start degrading and you need to rearchitect them. Very few problems at that scale can be solved by throwing more hardware at them.