Re:Show HN: Up to 100x Faster FastAPI with simdjson and io_u...
by AtlasBarfed on Monday March 06, 2023 @07:58PM
(#35050093)
Well, here's an idea. Do you HAVE to have the file as a big json file? How did it end up like that?
When you source the data, can you instead output the overall doc as "part" files that are aligned along subdocument boundaries?
THen you don't have to solve that "find the subdocument terminator" on stream-back.
You are PROBABLY building the json doc via some DB dump, so just change the code that dumps the data.
If you need to reconstruct it as a single json doc for some reason, it is about as trivial as it gets to reconstruct the data as a single document.
Or programattically, a "file" object that just does the reconstruction on the fly from the source part files as it is read, that should be relatively trivial in lots of languages.
When you source the data, can you instead output the overall doc as "part" files that are aligned along subdocument boundaries?
THen you don't have to solve that "find the subdocument terminator" on stream-back.
You are PROBABLY building the json doc via some DB dump, so just change the code that dumps the data.
If you need to reconstruct it as a single json doc for some reason, it is about as trivial as it gets to reconstruct the data as a single document.
Or programattically, a "file" object that just does the reconstruction on the fly from the source part files as it is read, that should be relatively trivial in lots of languages.