News for hackers. Stuff that matters.
Re:Parsing JSON Is a Minefield (2016) by EdwardDiego on Monday October 11, 2021 @09:15AM (#28827974)
> How many JSON nesting levels does your implementation allow? What happens if it's exceeded

Haha, I've met a few stack overflows in this area.

Re:Parsing JSON Is a Minefield (2016) by thechao on Monday October 11, 2021 @04:12PM (#28832493)
I had a little non-recursive JSON parser hanging around. When you have "nested" levels you've really only got two choices: object, or array. That implies that to track nesting, you just need an array of 1b values. In order to shave the yak properly, I built "nesting compressor" that detected runs of array/object and represented them using a 64b RLE; or, it bailed out, and then just used on-the-fly compression with zstd.

Obviously, any sort of JSON file that fit on a disk I can afford can be parsed into memory in a tiny fraction of its on-disk representation. I modified `yes` to just stream `[` out; the JSON parser handled it just fine — it takes a while to roll a 64b counter.

Re:Parsing JSON Is a Minefield (2016) by tehbeard on Monday October 11, 2021 @09:54AM (#28828322)
While there's a lot of issues with JSON, this one also applies to any other interchange format that supports nesting, including the much beloved XML. Protobuf might also have this, idk if it does any static analysis for infinite depth.
Re:Parsing JSON Is a Minefield (2016) by q3k on Monday October 11, 2021 @10:10AM (#28828479)
The problem doesn't really exist in Protobuf, as protobuf (de)serialization is performed based on an IDL definition of the message type. Whatever that IDL specifies, a corresponding typed definition and (de)serialization function will be generated for your programming language, and that implementation will ignore any fields that weren't part of the IDL. The (de)serializing code is statically generated ahead of time, and is treated like any other code that operates on potentially nested data structures.

What this means is that if your IDL specifies deep nesting (or recursive nesting), then it means your application is expected to handle this “by contract”, and attempts to deserialize will rightfully fail in case of out-of-memory / stack overflow errors. There's no danger of an implementation 'accidentally' deserializing something nested that was passsed from the outside, as anything unknown to the IDL is simply ignored.

Finally, there's no XML-like self-references in Protobuf, so it's not possible to have an infinitely deep structure, or a combinatorial explosion like with billion laughs - just a very deeply nested one, and only if allowed in the IDL, and only up to whatever message size limit you're allowing.

Re:Parsing JSON Is a Minefield (2016) by tehbeard on Monday October 11, 2021 @10:53AM (#28828980)
Thank you for the 2nd + 3rd paragraphs, those were parts of protobufs design I wasn't really aware of from a cursory glance.

I'm a little suprised to learn there's no self-reference support in protobuf, as I wouldn't have assumed parsing that would be an issue (as all it really is is a pointer to an existing object in the message to say, put a ref. to it here), though I guess it might be a problem in supporting certain languages.

Re:Parsing JSON Is a Minefield (2016) by q3k on Monday October 11, 2021 @11:00AM (#28829070)
> I'm a little suprised to learn there's no self-reference support in protobuf, as I wouldn't have assumed parsing that would be an issue (as all it really is is a pointer to an existing object in the message to say, put a ref. to it here), though I guess it might be a problem in supporting certain languages.

That's a tradeoff more designs should have, IMO: reduce the feature set as much as possible, but in return make the implementation vastly simpler. :)

I assume it's not only about support in programming languages, but also exactly to eliminate the entire class of bugs that stems from back/forward-references in serialized data, and to generally keep the wire format as simple (to parse and to implement a parser for) as possible. The few usecases that could make use of references are not worth the pain inflicted on everyone if they were implemented.

Re:Parsing JSON Is a Minefield (2016) by charleslmunger on Monday October 11, 2021 @07:10PM (#28834143)
To clarify, tags unknown to the IDL are propagated without parsing, not ignored.
Re:Parsing JSON Is a Minefield (2016) by ChrisMarshallNY on Monday October 11, 2021 @11:07AM (#28829149)
> the much beloved XML

I can't quite resolve "beloved" and "XML" in the same sentence...

That said, I have used XML a lot, pretty much because of XML Schema.

I don't like it. No sir. Not one bit. Uh-uh...

But there's really no viable substitute.

When I design an API, I generally start with an object model, and use native converters to create XML and JSON from it.

I will provide an XML Schema with the XML variant. I often have to do this by hand, which sucks. There are tools to create Schema from dumps, but these are pretty limited. I may use them to "get me in the ballpark," but there's always lots of elbow grease.

I'll use the XML for testing, but I will usually use the JSON format for the actual implementation.

Re:Parsing JSON Is a Minefield (2016) by nocman on Monday October 11, 2021 @01:23PM (#28830674)
> I can't quite resolve "beloved" and "XML" in the same sentence...

You mean, it's possible to take that as being NON-sarcastic??? If so, I share your lack of resolution.