News for hackers. Stuff that matters.
Re:PyPy's New JSON Parser by umvi on Tuesday October 08, 2019 @11:32AM (#21193387)
Hmm, I'm starting to think my definition of "small" is different from everybody else's. I consider "smallish" JSON to be at most 100KB (and on average 1-10 KB). 1 MB I consider to be "large"... basically anything that doesn't parse near instantaneously I consider "large".
Re:PyPy's New JSON Parser by Uehreka on Tuesday October 08, 2019 @12:20PM (#21193916)
I think GP might've been saying 1M small JSON documents, rather than a single document containing 1M strings.
Re:PyPy's New JSON Parser by nicoburns on Tuesday October 08, 2019 @01:58PM (#21195058)
If you use a language like Rust (which is really easy for these kind of jobs), then 1mb will parse almost instantly.

I've been working with a 90gb JSON file recently. I'd probably start counting a file as large at around 100mb.

Re:PyPy's New JSON Parser by aisengard on Tuesday October 08, 2019 @02:15PM (#21195258)
I have to ask - why? I understand if you're dealing with an external resource that you can't change, but I'd still like to ask them the same question. JSON's strength, it seems to me, is that it's great for human consumption. It's a very readable data structure when serialized! But who is ever going to read 90GB of JSON?
Re:PyPy's New JSON Parser by nicoburns on Tuesday October 08, 2019 @05:44PM (#21197634)
Not out of choice! It was a backup of a firebase database (also not my choice). I totally agree that JSON is not a good format for files this large, but it was the only format the data was available in so what can you do.
Re:PyPy's New JSON Parser by nly on Tuesday October 08, 2019 @02:33PM (#21195473)
Not OP, but I deal with a variety of proprietary binary/encoded files in the 10s of MB to the 10s of GB range at work, and we have started decoding to JSON as the first step in our normalization, indexing and analytics pipeline (To be honest we essentially just throw everything in to MongoDB). Daily we're in the terabyte range.

It's the perfect solution because you can use the same decoder + something like jq for debugging and manipulation on the command line.

Re:PyPy's New JSON Parser by e12e on Tuesday October 08, 2019 @03:38PM (#21196287)
A question - do know of any good resources for jq? I haven't quite been "angry enough" to use jq much beyond using it for formatting json - but that's also partly because I can't seem to quite grasp the syntax.

It isn't quite "awk-ish", not quite cut/join/sort/uniq. But also not quite graphql? Which I suppose would be the "natural" language for a json-query tool?

I feel like I'm missing a piece - and missing out...

Re:PyPy's New JSON Parser by chucky_z on Wednesday October 09, 2019 @12:43AM (#21200098)
The manual is pretty dang good, tbh.

https://stedolan.github.io/jq/manual/

Re:PyPy's New JSON Parser by e12e on Wednesday October 09, 2019 @02:26AM (#21200604)
I agree that it is clear, but I still struggle to figure out (by myself) how to do (in an awk sense) simple things like:

Filter out empty or null values:

https://stackoverflow.com/questions/39500608/remove-all-null...

I seem to either end up with unwieldy, long/verbose solutions - or shorter ones that does not work :)

Re:PyPy's New JSON Parser by chucky_z on Thursday October 10, 2019 @10:48PM (#21221111)
Oh wow, that's terrible!

I tend to get 90% of the way with jq and then awk/sed/grep my way to the desired result (jq -r).

Re:PyPy's New JSON Parser by orf on Tuesday October 08, 2019 @02:40PM (#21195567)
It's not necessarily that it's great for human consumption, more that it's super interchangeable and ridiculously easy to get up and running, and for the most part 90Gb of space is so cheap that it's not worth the overhead of using a binary format.

I'm guessing that OP wasn't parsing a single 90Gb object, but a stream of many newline delimited JSON objects.

Plus the human element helps.

Re:PyPy's New JSON Parser by nicoburns on Tuesday October 08, 2019 @05:43PM (#21197627)
> I'm guessing that OP wasn't parsing a single 90Gb object, but a stream of many newline delimited JSON objects.

I wish that were true. Alas it really was a 90gb object. Actually, the first thing I did was split it by top-level key before processing further.

Re:PyPy's New JSON Parser by weberc2 on Tuesday October 08, 2019 @02:50PM (#21195697)
Especially considering that JSON compresses well.
Re:PyPy's New JSON Parser by ken on Tuesday October 08, 2019 @01:23PM (#21194632)
Isn't that tautological? Clearly, no JSON parser provides a significant gain on parse speed of small JSON documents, when you define large as those where parse time is significant.
Re:PyPy's New JSON Parser by coldtea on Tuesday October 08, 2019 @03:55PM (#21196476)
>Isn't that tautological?

No, it's no.

>Clearly, no JSON parser provides a significant gain on parse speed of small JSON documents, when you define large as those where parse time is significant.

A rather bizarre deduction.

E.g. "define large as those where parse time is significant".

So let's say a large file is one whose parse time is > 10 seconds.

So, while looking at small, ie. < 10 second parse time files, if a JSON parser takes files that others load in 500ms and load them in 20ms, then that's "a significant gain on parse speed of small JSON documents"...

We didn't have to change any of the definitions...

Re:PyPy's New JSON Parser by umvi on Tuesday October 08, 2019 @01:56PM (#21195038)
Well, Python2 -> Python3 introduced a dramatic performance increase, even for small JSON documents.
Re:PyPy's New JSON Parser by IanCal on Tuesday October 08, 2019 @03:37PM (#21196271)
These were small though, I just made a small sample of a million of them and still saw a 2x improvement.
Re:PyPy's New JSON Parser by IanCal on Tuesday October 08, 2019 @03:36PM (#21196259)
Oh I think you misunderstood, they are all small, there's just a million of them. I think they were all about 1k.