Half a TB per node, which during regular compaction can double. And if you went over, your CPU and disk spent so much time on overhead such as JVM garbage collection that your compaction processes backlog, your node goes slower and slower, your disk eventually fills up, and it falls over. Later things got better and you could use bigger nodes if you knew what you were doing and didn't trip over any of the hidden bottlenecks in your workload. Maybe even fixed in the last few versions of Cassandra 3x and 4.0.
Past 100k servers you start needing really intense automation just to keep the fleet up with enough spares.
If you’ve got say 10k servers it’s much more manageable
The fun thing is Cassandra was born at FB but they don’t run any Cassandra clusters there anymore. You can use lots of cheap boxes but at some point the failure rate of using soo many boxes ends up killing the savings and the teams.
Isn't Intragram mostly Cassandra?
https://instagram-engineering.com/open-sourcing-a-10x-reduct...
Zippydb is honestly one of the best parts of fb infra. It let you select levels of consistency vs latency
How is that different from Cassandra's Tunable consistency model?
https://cassandra.apache.org/doc/4.1/cassandra/architecture/...
Apple is using less than one TB per server…
But when you see the 1000s of clusters it starts to make sense. They probably have a Cassandra cluster as their default storage for any use case and each one probably requires at least 3 nodes. They’re keeping the blast radius small of any issue while being super redundant. It probably grew organically instead of any central capacity management