Re:A Journey building a fast JSON parser and full JSONPath, ...
by beached_whale on Sunday July 05, 2020 @01:16PM
(#23740235)
Are there benchmarks with simdjson getting the JSON reified into native data structures outside the parser?
Re:A Journey building a fast JSON parser and full JSONPath, ...
by glangdale on Sunday July 05, 2020 @06:08PM
(#23742518)
No. This is an interesting problem but a non-goal for the C++ version of simdjson, which produces usable "native data structures" albeit not outside the parser (speaking informally - the structures aren't really "in the parser" but they are specific to simdjson if you see what I mean).
The reification problem is awkward but it's a different problem for each language. I did daydream of breaking some API boundaries and using SIMD to magic up some Python structures directly, but this would be incredibly brittle. Someone would probably use it but the chance of getting burned by this is... high.
Re:A Journey building a fast JSON parser and full JSONPath, ...
by jkeiser on Sunday July 05, 2020 @06:13PM
(#23742562)
We have started experimenting how to do it with simdjson, though :) https://github.com/simdjson/simdjson/pull/947
Re:A Journey building a fast JSON parser and full JSONPath, ...
by beached_whale on Sunday July 05, 2020 @06:22PM
(#23742618)
Sorry, I meant the applications data structures and not the libraries. And C++ yes, at least that is my exposure to simdjson.
"Informal tests", especially ones with zero description as to what was being tested or even ballpark numbers as to how fast this system supposedly run, may not entirely address this point.
We wrote a paper and anyone curious about our methodology can download and run simdjson with a few clicks and verify our performance claims, which (I hope) are well-described in the paper and line up with the arguments we make in favour of the design.
Given that no indication of what was being measured is provided, it's kind of hard to tell what this performance claim even is, or how to verify it.