ParaText: CSV parsing at 2.5 GB per second

Re:ParaText: CSV parsing at 2.5 GB per second by burntsushi on Wednesday June 08, 2016 @07:53AM (#11861554)

From looking at their source briefly, I can make two observations:

Firstly, their core parsing routine is a state machine not unlike the one found in rust-csv: https://github.com/wiseio/paratext/blob/master/src/csv/rowba...

Secondly, they do indeed appear to be achieving high parsing speed using coarsely grained parallelism. You can see the code that chunks up the CSV data here: https://github.com/wiseio/paratext/blob/master/src/generic/c...

It's pretty clever!

[ Parent | on HN ]

Re:ParaText: CSV parsing at 2.5 GB per second by Ygg2 on Wednesday June 08, 2016 @11:07AM (#11862711)

Can this be used to speed up rust-csv?

[ Parent | on HN ]

Re:ParaText: CSV parsing at 2.5 GB per second by burntsushi on Wednesday June 08, 2016 @04:28PM (#11865390)

It feels like something that should be built on top of rust-csv.

[ Parent | on HN ]

Re:ParaText: CSV parsing at 2.5 GB per second by Ygg2 on Thursday June 09, 2016 @07:45AM (#11868736)

I thought this was base ability of the parser to parse CSV in chunks.

[ Parent | on HN ]