Re:Odo: Shapeshifting for your data
by nl on Wednesday January 04, 2017 @12:41AM
(#13315931)
(Py)Spark? I run it locally and do that kind of operation pretty frequently.
[ Parent
| on HN ]
On what scale?
Part of the problem is the scale, but another part is that writing partitioned parquets seems poorly documented (I would love corrections, I spent a decent amount of time last week looking for good information)
I was recently working with a 38GB compressed text dataset locally. I can't remember how big it was in parquet.
It's literally one line of code. See http://labs.vistarmedia.com/2016/12/27/indexing-json-logs-wi... for an example (except you write to a local file system rather than HDFS).