Re:Analyzing multi-gigabyte JSON files locally
by aardvarkr on Saturday March 18, 2023 @11:53PM
(#35215877)
Have you implemented this idea in code? Seems interesting and worth playing around with
Yes, the code is linked in the paper: https://github.com/ot/semi_index
Several years ago I wrote a paper [1] on representing the parse tree of a JSON document in a tiny fraction of the JSON size itself, using succinct data structures. The representation could be built with a single pass of the JSON, and basically constant additional memory.
The idea was to pre-process the JSON and then save the parse tree, so it could be kept in memory over several passes of the JSON data (which may not fit in memory), avoiding to re-do the parsing work on each pass.
I don't think I've seen this idea used anywhere, but I still wonder if it could have applications :)
[1] http://groups.di.unipi.it/~ottavian/files/semi_index_cikm.pd...