Simdjson – Parsing Gigabytes of JSON per Second

Re:Simdjson – Parsing Gigabytes of JSON per Second by xnormal on Thursday February 21, 2019 @02:34AM (#19215202)

Any chance of something similar for CSV? (full RFC-4180 including quotes, escaping etc).

Terabytes of "big data" get passed around as CSV.

[ Parent | on HN ]

Re:Simdjson – Parsing Gigabytes of JSON per Second by glangdale on Thursday February 21, 2019 @03:00AM (#19215290)

CSV is on our list; this is a simpler task than JSON due to the absence of arbitrary nesting.

[ Parent | on HN ]

Re:Simdjson – Parsing Gigabytes of JSON per Second by imtringued on Thursday February 21, 2019 @04:28AM (#19215666)

I doubt someone using CSV for big data is going to follow that rule...

[ Parent | on HN ]

Re:Simdjson – Parsing Gigabytes of JSON per Second by carlmr on Thursday February 21, 2019 @06:26AM (#19216073)

What do you mean? It's not a rule, it's just not possible in the CSV format to have arbitrary nesting.

[ Parent | on HN ]

Re:Simdjson – Parsing Gigabytes of JSON per Second by blaisio on Thursday February 21, 2019 @04:44AM (#19215732)

It's probably relevant to mention https://github.com/BurntSushi/rust-csv. It uses a state machine (which seems to be the author's expertise) to parse CSVs really fast. Based on some other work, you can do better if you use some of the new SIMD instructions.

[ Parent | on HN ]

Re:Simdjson – Parsing Gigabytes of JSON per Second by badeu on Thursday February 21, 2019 @04:32AM (#19215682)

I've developped a full RFC compliant CSV parser with Python bindings and supporting SSE4 to AVX-512 instruction sets, however i'm struggling with my hierarchy to open-source it at the moment.

But, the goal of my message is not to tease you with an unavailable code. It's just to say it is a lot more simpler to write a CSV parser than a JSON parser.

So, do not hesitate to write one yourself ! It's easy and a nice way to introduce yourself to SIMD instructions.

[ Parent | on HN ]