There's also the argument that businesses hoping to scale might be better off assuming they will need it to save themselves work down the road.
Your Data Isn't That Big https://www.chrisstucchio.com/blog/2013/hadoop_hatred.html
This is an extremely important lesson. You don't have Big Data, unless you're one of a handful of huge companies.
Years ago I joined a mid-size public company and inherited one particular service. Management was adamant that it had to migrate away from MySQL onto Cassandra due to data volume. But there were no metrics being tracked, it was driven by "we have to do it because everyone is doing it".
I started tracking actual usage, growth and performance and it didn't seem to point to a problem that needed solving. So I halted that project (to their credit, they listened to engineering input so I was able to).
A few years later when I left, it was still perfectly fine on MySQL. Nowhere close to running out of capacity. Today it's been almost ten years and while I'm not there, what I hear is that it's still fine.
My point is that large companies that do need to scale hire many developers. FAANG likely hire north of several hundred thousand engineers alone, and according to this article[1] there are only something like 1.5 million software engineers in the US.
[1]: https://www.daxx.com/blog/development-trends/number-software...
Of course that's a oversimplification, but the data generated today is mind-boggling large compared to the 90s.
In most cases, you have colossally fewer than 1M writes per second, and you usually distribute them over several boxes. the IoT avalanche of writes in particular is naturally sharded, that is, all these writes can occur independently, and then you can replicate statistics into an analytical RDBMS.
It's a large write load that needs to be transactionally ordered, like a stream of purchases in a large online store, that can be a hard(er) problem for an RDBMS.
My example was just one use-case where one should consider using a more unconventional DBs like Cassandra or HBase.
For what percentage of companies would you say your statement is true?