Re:Parallel Out-Of-Core Dataframes in Python: Dask and OpenS...
by mrocklin on Sunday August 16, 2015 @10:18AM
(#10068947)
Flat CSV or JSON files are hard to parse. Fast CSV parsers and gzip decompression both run at around 100MB/s. If you want to get faster than this you'll need to use better (ideally binary) formats.
This notebook might interest you: http://nbviewer.ipython.org/gist/mrocklin/c16c5c483b2b9859de... , particularly the sections starting at "Eleven minutes is a long time." It compares CSV costs (minutes) to custom binary storage formats (seconds) on a 20 GB dataset.
I've been looking into tools like Spark and Drill, but my tests running on a single machine found them to be extremely slow. Maybe things would be faster if I converted the log files to their native formats?
I've been considering trying to load the data into a postgres db using cstore_fdw, but what I really just want is a high performance sql engine for flat files, something probably like Kdb.
Like this article that I read recently: http://www.frankmcsherry.org/graph/scalability/cost/2015/01/... I know this can be done efficiently enough on a single machine, I just need the right software.