ParaText: CSV parsing at 2.5 GB per second

Re:ParaText: CSV parsing at 2.5 GB per second by ygra on Wednesday June 08, 2016 @12:25AM (#11860012)

Does cut work with quoted fields correctly? My understanding was that it was just a dumb line tokenizer and CSV is a little more complex than that, e.g., CSV rows can span more than one line when fields contain line breaks.

[ Parent | on HN ]

Re:ParaText: CSV parsing at 2.5 GB per second by gkst on Wednesday June 08, 2016 @07:02AM (#11861370)

No, it doesn't and therefore is no viable option for many real-world datasets.

[ Parent | on HN ]

Re:ParaText: CSV parsing at 2.5 GB per second by rspeer on Wednesday June 08, 2016 @12:28AM (#11860023)

Which means an effective thing to do when you control the data pipeline is to ban tabs and line-breaks from your values, and then use `cut` on tab-separated files.

[ Parent | on HN ]

Re:ParaText: CSV parsing at 2.5 GB per second by ygra on Wednesday June 08, 2016 @02:24AM (#11860382)

Sure, if you can redefine your data format, you can take shortcuts. But otherwise comparing cut to a CSV parser is just a different way of saying »if it doesn't have to be correct, I can make it as fast as I want«.

[ Parent | on HN ]

Re:ParaText: CSV parsing at 2.5 GB per second by Twirrim on Wednesday June 08, 2016 @01:04AM (#11860144)

Or just not use CSV as a data transfer format :D

[ Parent | on HN ]

Re:ParaText: CSV parsing at 2.5 GB per second by rspeer on Wednesday June 08, 2016 @11:52AM (#11863060)

I think you're missing the point. Unix tools like `cut` are really effective ways to deal with plain text data, who cares if it's CSV.

[ Parent | on HN ]