Re:Sonic: A fast JSON serializing and deserializing library
by Groxx on Tuesday November 23, 2021 @06:09PM
(#29324716)
Depends. I've had multi-gigabyte `[{..}, {..}, ...]` json arrays from database dumps, and doing even basic things with that with jq takes ages unless you use the (highly obtuse IMO) streaming methods. Sometimes you can pre-grep to filter the results to something trivial to process, but sometimes the structure is not unique enough to let you do that, or it depends on multiple field values - filtering that with a json parser makes perfect sense, and then speed can matter.
That said, a 2x+ improvement for a couple megabytes, especially if done many times per second, is still a significant improvement.
Re:Sonic: A fast JSON serializing and deserializing library
by hobs on Tuesday November 23, 2021 @11:28PM
(#29326925)
Monthly general ledger entries for the largest real estate companies, tried XML and JSON, eventually landed on compressed CSV for best trade off between human readable large files (~1-3GB) and compressibility.
Re:Sonic: A fast JSON serializing and deserializing library
by jeffbee on Tuesday November 23, 2021 @06:10PM
(#29324727)
GeoJSON data can be just as big as you’d like. I have a big archive of GIS data where the files are enormous.
Re:Sonic: A fast JSON serializing and deserializing library
by magicalhippo on Tuesday November 23, 2021 @07:29PM
(#29325512)
Not JSON, but we process XMLs on the order of a GB. Largest ones are consolidated invoices (ie a lot of separate invoices in one file). Other large ones contain rules and codelists in multiple languages.
Re:Sonic: A fast JSON serializing and deserializing library
by jonstewart on Tuesday November 23, 2021 @07:39PM
(#29325591)
All the time. It's not there's a single record that is that size, but all sorts of things log in json, so you wind up with multi-GB jsonl files. As an example: AWS CloudTrail logs.
Re:Sonic: A fast JSON serializing and deserializing library
by nerdponx on Tuesday November 23, 2021 @06:10PM
(#29324723)
Huge log files are one use case. Large datasets are another, in data analysis, machine learning, and ETL tasks.
I love the idea of a library trying to squeeze every last bit of performance out of the CPU, but I'm genuinely curious at the problems it solves in the real world.