News for hackers. Stuff that matters.
Re:Daft: Distributed DataFrame for Python by jmakov on Friday March 01, 2024 @03:30AM (#39559802)
Wondering if network becomes a bottleneck. Anybody using this?
Re:Daft: Distributed DataFrame for Python by teitoklien on Friday March 01, 2024 @07:25AM (#39561107)
Great question!

Depends on your workload size, if the compute time is less than time it takes to transfer the data back and forth, then it might be a bad idea to use it indeed.

Typically these solutions should only be tested after maxing out vertical scaling, before applying for horizontal scaling.

Network is one hell of a destroyer when it comes to advantages gained from distributed computing.

Re:Daft: Distributed DataFrame for Python by spratzt on Friday March 01, 2024 @10:47AM (#39562841)
I would go even further and argue that for the vast majority of organizations vertical scaling is all you are ever going to need.
Re:Daft: Distributed DataFrame for Python by sammysidhu on Friday March 01, 2024 @12:24PM (#39564103)
horizontal scaling however provides you with more aggregate network bandwidth. Most enterprises run workloads that downloads data from a data lake (usually S3), which is usually the bottleneck. Having horizontal scaling here allows the query leverage much higher network than just having a large single machine.
Re:Daft: Distributed DataFrame for Python by jaychia on Friday March 01, 2024 @11:19AM (#39563235)
Hello, Daft developer here!

The network indeed becomes the bottleneck. In 2 main ways:

1. Reading data from cloud storage is very expensive. Here’s a blogpost where we talk about some of the optimizations we’ve done in that area: https://blog.getdaft.io/p/announcing-daft-02-10x-faster-io

2. During a global shuffle stage (e.g. sorts, joins, aggregations) network transfer of data between nodes becomes the bottleneck.

This is why the advice is often to stick with a local solution such as DuckDB, Polars or Pandas if you can keep vertically scaling!

However, horizontally scaling does have some advantages:

- Higher aggregate network bandwidth for performing I/O with storage

- Auto-scaling to your workload’s resource requirements

- Scaling to large workloads which may not fit on a single machine. This is more common in Daft usage because we also work with multimodal data such as images, tensors and more for ML data modalities.

Hope this helps!