Re:Analyzing multi-gigabyte JSON files locally
by saidinesh5 on Saturday March 18, 2023 @01:43PM
(#35211337)
There's libraries like ijson.
At my last job, i wrote a wrapper around this C++ library called rapidjson to parse streaming json. It worked quite well.
Here's the write up about it: https://dinesh.cloud/2022/streaming-json-for-fun-and-profit/ And here's the code: https://github.com/multiversal-ventures/json-buffet
The API isn't the best. I'd have preferred an iterator based solution as opposed to this callback based one. But we worked with what rapidjson gave us for the proof of concept. The reason for this specific implementation was we wanted to build an index to query the server directly about it's huge json files (compressed size of 20+GB per file) using http range queries.
Yeah but parsing it can require ~2x the RAM available and push you into swap / make it not possible.
> Or write a custom little program that streams data from the JSON file without buffering it all in memory. JSON parsing libraries are plentiful so this should not take a lot of code in your favorite language.
What is the state of SAX JSON parsing? I used yajl a long time ago but not sure if that’s still the state of the art (and it’s C interface was not the easiest to work with).