I tried to use Dask to do a formst conversion of a 5 TB binary dataset coming from astrophysical simulation. It should have been a trivial job, since it was a single huge 3D array coming from a lot of files that were slicing the array in one direction, and I needed to do a conversion in each point, do a couple of transposes and reshapes, and then output a single file.

After two weeks tearing my hair out at the appalling speed, I scrapped that solution and spent one day throwing together a pipeline of classical shell tools (dd, sed, xxd etc.), orchestrated using GNU Parallell, and it did the whole job in three hours.

Funny and I had a similar real life experience with a hadoop cluster. I replaced a job that needed 10 machines with a few lines of C, some python and GNU parallel, which was perl I believe at the time. The process was taking over 24 hours on the cluster and became unworkable. I got it back to 2 hours on my slow Macbook air.

On Dask, it may not be super fast for very large data sets but it is sure easy to use.

I'm not making any claims about dask being super reliable or easy to use, but let's not pretend that a spark cluster is a trivial thing to set up and deal with.

I've had to administer spark clusters in the past, and there's a good reason why databricks is a thing.

Another detail that's important is that spark is dramatically overkill for the data sets dask is meant to work on. Dask is really geared for data sets that can't fit in memory but not necessarily data sets that can't fit on disk that spark was really meant for.

> spark is dramatically overkill for the data sets dask is meant to work on

I disagree with this, I have used spark standalone/single node both locally and in the cloud for numerous use cases because you can easily cook up spark sql or pyspark cleaning and ETL scripts for small data (a couple TB) and reach high throughput (>260mb/s). I am also someone who reaches for dask (and prefect) first for small data but in cases where the data is medium sized (100gb-10tb) I always reach for spark first, its simply easier to write fast code with spark (especially spark sql) without needing to think hard about optimizations. I also have not run into any operational issues running a spark standalone instance for a few hours or days to process small data I would use dask for.

I reach for dask first for most small data (<100gb) because its better integrated with python libraries like prefect that improve the dev experience.

Spark seems like it will be good, but always disappoints.

I doubt know much about spark, so I may get this wrong, but I feel that they need to greatly improve both: 1) the automatic configuration of single node systems 2) the changing of config throughout the processing

The fact that I don't even really know if this is the problem however is the real problem. I get that setting stuff up for a huge network of computers may be challenging, but standard defaults to use all processors on a single machine should be automatic.

Spark pisses me off that the defaults for a 1TB RAM, 20 core machine as a single node can't handle reading a 20GB parquet file with one line. And further, checking substrings can throw OOM errors in the next step without adjusting the config of worker memory, etc.

Worst of all, a 40 MB (!) file after filtering and processing can take like 10 minutes to save because of all the parallelism config. It's off course instantaneous on pandas. They really need to fix the experience for single machines (e.g. just like a small 18 core, 512 GB ram system), so that people can write packages that work for anyone, but with a small yaml flag, it can be sent to a cluster.

On the SQL front there has been some active work to make that experience better with DASK.

See dask-sql: https://dask-sql.readthedocs.io/en/latest/pages/api.html

Can't you just run it in Docker with a one-liner? I don't think you need a whole cluster just to leverage the Spark tooling.

If you're in the JVM you can basically embed it in-process.

Yes. I'm not a fan of Spark, dealing with JVM, new syntax everything, optimizing parallelism in a weird way but - it always works.

Dask, on the other hand, works some of the time. The rest of the time it'll keep running a calculation forever, or simply fail silently over and over, or some other unpleasant outcome.

I had a similar experience, I usually avoid the dask dataframe where possible and instead use bag and dask delayed. But its hard to get scientists to give up the dataframe mindset. Thats one of the reasons I wanted to try modin, I have heard the dataframe in modin is a bit easier to use.

I was mainly looking at Dask for the purpose of a more better Pandas DataFrame, but it seems like it's actually not that much better, just more distributed. It's like Dask is trying to do too many things and there doesn't do most of them well enough.

While I haven't tried it, vaex[1] seems much better, since it specifically to only address one pain point, which is to address the inefficiencies of the pandas DataFrame and aims to do that one thing well.

[1] https://vaex.io/docs/index.html

Vaex is amazing, I can definitely recommend it.

So much this. Dask does not remove the need to babysit workers and partitions, and that's the real difference between pandas and spark.

It's been over a year now, but I had an identical experience.

a path I haven't explored yet is Koalas, which touted a 80% API compatibility with pandas at version 1.0. Now its at version 1.8, and it has the pandas API as the main design principle for function interface design.

https://databricks.com/blog/2020/06/24/introducing-koalas-1-...