Re:ParaText: CSV parsing at 2.5 GB per second
by falaki on Tuesday June 07, 2016 @08:20PM
(#11859133)
I wish there was more details on the benchmark. For example, I am interested to know if schema inference was turned on for spark-csv?
[ Parent
| on HN ]
Re:ParaText: CSV parsing at 2.5 GB per second
by gcommer on Tuesday June 07, 2016 @08:41PM
(#11859209)
The benchmarking scripts are in their repo: this line seems to answer your question: ("inferschema" was turned on, probably to make it an apples-to-apples comparison since static schemas is still on their TODO) https://github.com/wiseio/paratext/blob/ca347a552a53595b680c...