News for hackers. Stuff that matters.
Re:Odo: Shapeshifting for your data by eximius on Wednesday January 04, 2017 @12:33AM (#13315898)
I wish it supported large csv to partitioned parquet. THAT is something I need a good solution for.
Re:Odo: Shapeshifting for your data by quasiben on Wednesday January 04, 2017 @12:40AM (#13315927)
Are you familiar with fastparquet (https://github.com/dask/fastparquet) and pyarrow (https://pyarrow.readthedocs.io/en/latest) ?
Re:Odo: Shapeshifting for your data by eximius on Wednesday January 04, 2017 @02:06AM (#13316262)
Not until you linked me. :) They look like good steps but not quite what I need, for different reasons. I'll have to check when I get back to the office next week.
Re:Odo: Shapeshifting for your data by ColanR on Wednesday January 04, 2017 @12:44AM (#13315952)
Your link is malformed. https://github.com/dask/fastparquet
Re:Odo: Shapeshifting for your data by ColanR on Wednesday January 04, 2017 @12:44AM (#13315954)
...and its fixed.
Re:Odo: Shapeshifting for your data by nl on Wednesday January 04, 2017 @12:41AM (#13315931)
(Py)Spark? I run it locally and do that kind of operation pretty frequently.
Re:Odo: Shapeshifting for your data by eximius on Wednesday January 04, 2017 @02:13AM (#13316298)
On what scale?

Part of the problem is the scale, but another part is that writing partitioned parquets seems poorly documented (I would love corrections, I spent a decent amount of time last week looking for good information)

Re:Odo: Shapeshifting for your data by nl on Wednesday January 04, 2017 @10:43AM (#13319139)
I was recently working with a 38GB compressed text dataset locally. I can't remember how big it was in parquet.

It's literally one line of code. See http://labs.vistarmedia.com/2016/12/27/indexing-json-logs-wi... for an example (except you write to a local file system rather than HDFS).

Re:Odo: Shapeshifting for your data by eximius on Wednesday January 04, 2017 @10:52AM (#13319234)
That looks very helpful, I'll have to check it out!