I would go even further and argue that for the vast majority of organizations vertical scaling is all you are ever going to need.
Re:Daft: Distributed DataFrame for Python
by sammysidhu on Friday March 01, 2024 @12:24PM
(#39564103)
horizontal scaling however provides you with more aggregate network bandwidth. Most enterprises run workloads that downloads data from a data lake (usually S3), which is usually the bottleneck. Having horizontal scaling here allows the query leverage much higher network than just having a large single machine.
Depends on your workload size, if the compute time is less than time it takes to transfer the data back and forth, then it might be a bad idea to use it indeed.
Typically these solutions should only be tested after maxing out vertical scaling, before applying for horizontal scaling.
Network is one hell of a destroyer when it comes to advantages gained from distributed computing.