News for hackers. Stuff that matters.
Re:Daft: Distributed DataFrame for Python by teitoklien on Friday March 01, 2024 @07:25AM (#39561107)
Great question!

Depends on your workload size, if the compute time is less than time it takes to transfer the data back and forth, then it might be a bad idea to use it indeed.

Typically these solutions should only be tested after maxing out vertical scaling, before applying for horizontal scaling.

Network is one hell of a destroyer when it comes to advantages gained from distributed computing.

Re:Daft: Distributed DataFrame for Python by spratzt on Friday March 01, 2024 @10:47AM (#39562841)
I would go even further and argue that for the vast majority of organizations vertical scaling is all you are ever going to need.
Re:Daft: Distributed DataFrame for Python by sammysidhu on Friday March 01, 2024 @12:24PM (#39564103)
horizontal scaling however provides you with more aggregate network bandwidth. Most enterprises run workloads that downloads data from a data lake (usually S3), which is usually the bottleneck. Having horizontal scaling here allows the query leverage much higher network than just having a large single machine.