News for hackers. Stuff that matters.
Re:Analyzing multi-gigabyte JSON files locally by ot on Saturday March 18, 2023 @05:51PM (#35213450)
> Or write a custom little program that streams data from the JSON file without buffering it all in memory. JSON parsing libraries are plentiful so this should not take a lot of code in your favorite language.

Several years ago I wrote a paper [1] on representing the parse tree of a JSON document in a tiny fraction of the JSON size itself, using succinct data structures. The representation could be built with a single pass of the JSON, and basically constant additional memory.

The idea was to pre-process the JSON and then save the parse tree, so it could be kept in memory over several passes of the JSON data (which may not fit in memory), avoiding to re-do the parsing work on each pass.

I don't think I've seen this idea used anywhere, but I still wonder if it could have applications :)

[1] http://groups.di.unipi.it/~ottavian/files/semi_index_cikm.pd...

Re:Analyzing multi-gigabyte JSON files locally by aardvarkr on Saturday March 18, 2023 @11:53PM (#35215877)
Have you implemented this idea in code? Seems interesting and worth playing around with
Re:Analyzing multi-gigabyte JSON files locally by ot on Sunday March 19, 2023 @05:35AM (#35217416)
Yes, the code is linked in the paper: https://github.com/ot/semi_index