Re:Analyzing multi-gigabyte JSON files locally
by mindslight on Saturday March 18, 2023 @01:41PM
(#35211309)
* * *
[ Parent
| on HN ]
Re:Analyzing multi-gigabyte JSON files locally
by DonHopkins on Sunday March 19, 2023 @03:11AM
(#35216752)
You still have to pay for a 64 bit pointer to an un-interned string though. It's the fact that the keys are repeated for each record that overwhelms the size of the keys or pointers themselves, when you have millions of records. If you are using fixed structures, then the keys are implicit and don't need to be repeated or even represented as pointers, just known fixed offsets into the structures.
I wrote this about "Representing and Editing JSON with Spreadsheets":
https://news.ycombinator.com/item?id=21109798
https://medium.com/@donhopkins/representing-and-editing-json...
SpreadOn: Apply Directly to the Forehead.