If you have any code examples of either of those options I'd love to see them!
Edit: Also, I'm curious: what are the downsides to doing the mmap route?
You can mitigate this somewhat with explicit MAP_POPULATE and mlock.
They're... kinda free (in terms of capacity). A PTE or PDE is 64 bits per page. Even with 4kB pages, this is 8/4096 = 0.2% overhead. Churning the page tables and TLB as you walk the file is expensive, or at least historically was expensive (used to be that removing an entry required flushing the entire TLB; I don't think that's true anymore).
If you can use MAP_HUGE_2MB, this drops off to 0.00038% and the TLB impact goes way down.
> no way of handling errors.
Yeah, this aspect is really significant. Thanks for mentioning it.
When you source the data, can you instead output the overall doc as "part" files that are aligned along subdocument boundaries?
THen you don't have to solve that "find the subdocument terminator" on stream-back.
You are PROBABLY building the json doc via some DB dump, so just change the code that dumps the data.
If you need to reconstruct it as a single json doc for some reason, it is about as trivial as it gets to reconstruct the data as a single document.
Or programattically, a "file" object that just does the reconstruction on the fly from the source part files as it is read, that should be relatively trivial in lots of languages.
Simple - memory-map the whole file with `mmap`. We also suggest using `madvice`, to inform the kernel that you will be accessing the data just once, strictly in the sequential order. And then simply give it to SIMDJSON. With `ondemand` parser, your memory usage will be proportional to the depth of the deepest tree, not their number.
Harder way, would be to parse file in chunks, locating continuous sequences that form a single document.
There may be more options, but I'd have to check the docs :)