Faster and simpler with the command line: deep-comparing JSON files with jq

Re:Faster and simpler with the command line: deep-comparing ... by theamk on Thursday December 06, 2018 @11:35AM (#18619482)

(1) Try a language with fast allocations (C, C++, Rust, maybe Go or Java) -- anything except Python or Ruby

(2) Try using streaming API (I don't know Ruby, but quick google found https://github.com/dgraham/json-stream ). Note that this method will require you to massively restructure your program -- you want to avoid having all of the data in memory at once.

The streaming API might work better with jq-based preprocessing -- for example, if you want to compare two unsorted sets, it may be faster to sort them using jq, then compare line-by-line using streaming API.

[ Parent | on HN ]

Re:Faster and simpler with the command line: deep-comparing ... by dith3r on Thursday December 06, 2018 @12:02PM (#18619763)

Python is fast at parsing JSON, Go had hard time to match parsing speed of it. Additionally you have PyPy to help.

[ Parent | on HN ]

Re:Faster and simpler with the command line: deep-comparing ... by mattbillenstein on Thursday December 06, 2018 @12:20PM (#18620004)

Nod. https://gist.github.com/mattbillenstein/34cf2907390102ffbabd...

[ Parent | on HN ]

Re:Faster and simpler with the command line: deep-comparing ... by Filligree on Friday December 07, 2018 @09:59AM (#18628161)

Python is fast at doing anything that doesn't involve running Python.

That's an important caveat. Python's C JSON parser library is super-fast, but if you want to use the data for anything but a simple equality check afterwards, it'll be slow as molasses.

Or you'll write a C extension for it...

[ Parent | on HN ]