I assume you're mainly referring to buffer overflows, which are a problem with text based formats too. See for example the series of overflow vulnerabilities in IIS's HTTP parser which lead to some of the most disruptive worms in history like Code Red. Really this is more of a problem with memory-unsafe languages than serialization formats.
> Let's look at floating point numbers: with a binary format you can transmit NaN, Infinity, -infinity, and -0
Depending on the use case being able to encode these values may be a requirement, in which case binary is no worse than text.
> You can also create two NaN numbers that do not have the same binary representation.
This is specific to IEEE 754, not all binary representations have this issue. Text based formats also have far more pervasive problems with lacking a canonical representation so it's hard to count this as a point against binary.
> JSON is one of many competitors within an ecology of programming, including binary formats, and yet JSON currently dominates large parts of that ecology.
This is just an appeal to popularity fallacy.
That is the most visible security issue, but there are many others e.g. reading excess data is a security flaw (a la heartbleed).
> Really this is more of a problem with memory-unsafe languages than serialization formats
JSON is often used to communicate with unsafe languages. Are you suggesting a binary format is better for using with a language that is memory-unsafe? Or are your implying we should use Rust so that we can use a binary format!
> This is specific to IEEE 754, not all binary representations have this issue.
So now we use some other (unspecified) binary format for floating point numbers?
> This is just an appeal to popularity fallacy.
Bullshit. JSON didn't become popular because it was popular. Developers have chosen to use JSON because it served a purpose for them within their particular ecology. It became popular into the headwinds of XML and other formats.
There are plenty of unpopular binary formats that developers have had experience with that they choose not to use. I have personally have enough experience with a variety of spec and custom binary formats to know when I would use one.
Binary formats most definitely have their place (embedded, high throughput at scale, severe bandwidth restrictions, strict typing, languages with poor string handling, naturally binary data). But for a large percentage of software projects, JSON works and it works well.
XML became popular because of its human readability compared to binary and allowed disparate systems to cooperate. JSON is even more readable and allowed the whole XLST thing to be ignored, making a dev's life easier, and really took off with node.
Binary is not gone, but you don't see a lot of it in the 'web world', because everyone in that space is JSON, almost exclusively.
Having written many parsers, binary is by far and away the easiest: provided your language/environment supports it. That is, binary in js is a pain, because you don't have native ints and floats. In js, JSON is the base atomic data structure, so it makes complete sense that it's used... But in c/c++ and co. JSON is harder and introduces a lot of overhead/ quirks which simply don't exist in binary (provided you cover the buffer overruns and co.)
XML was simply never designed to represent structured data. It was meant to represent document markup in a way that was both simpler and more extensible than SGML.
If there's one thing the industry always seems to do, its embrace some technology hammer as the solution for every problem. Before XML some people were trying to trade around relational database dumps as data interchange formats, because relational databases were the golden hammer.
JSON is being abused by being stretched beyond its sweet spot as well, but there aren't industry consortiums necessarily pushing bad ideas like there was with SOAP and WS-*.
If you're missing anything out of WS-* in JSON, you can be sure somebody is working on a spec for it.
decompression and PNG libraries for example have caused massive security impact across the industry because of reuse in different products. Font handling, compressed bitmap, and windows cursor parsing also have been sources of issues.
Mozilla didn't just invest in Rust because parsing HTML and JSON are hard. Its all hard.
And then you go on to give examples of said scenarios of how this is true while saying I’m wrong? Anytime you have an unknown payload you have to make a determination of how long you’re going to wait, how much you’re going to accept, buffer, etc before it’s become a drain on the system
These are also easier to recover the data from in case of corruption, or re-sync the receiver, since you have a control byte you can sync on.
Also sometimes it is a feature. The payload of a NaN value can be user defined and some programs use it[1]. The string "NaN" drops information that might be usefull to some programs, it just doesn't affect as many as null -> "null" does.
[1] https://github.com/WebKit/webkit/blob/master/Source/JavaScri...
JSON is perfectly capable of representing integers which cannot be represented in IEEE-754 double precision floating point. That seems at least a little special to me.
Boy you are really stretching to make this sound complicated. It's not. You transmit 4 bytes or 8 bytes. Serialization is a memcpy().
You don't have to think about NaNs and Infinities because they Just Work -- unlike with textual formats where you need to have special representations for them and you have to worry about whether you are possibly losing data by dropping those NaN bits. If you want to drop the NaN bits in a binary format, it's another one-liner to do so.
It's funny that you choose to pick on floating-point numbers here, because converting floating-points to decimal text and back is insanely complicated. One of the best-known implementations of converting FP to text is dtoa(), based on the paper (yes, a whole paper) called "How to Print Floating-Point Numbers Accurately". Here's the code:
http://www.netlib.org/fp/dtoa.c
Go take a look. I'll wait.
dtoa() is not even the state of the art anymore. Just in the last few years there have been significant advances, e.g. Grisu2, Grisu3, and Dragon4...
Again, in binary formats, all that is replaced by a memcpy() of 4 or 8 bytes.
(A previous rant of mine on this subject: https://news.ycombinator.com/item?id=17277560 )
> > Length-prefixed binary formats are almost trivial to parse
> They definitely are not, as displayed by the fact that binary lengths are the root cause of a huge number of security flaws. JSON mostly avoids that.
Injection (forgetting to escape embedded text) is the root cause of a huge number of security flaws for text formats. Length-prefixed formats do not suffer from this.
What "huge number of security flaws" are you referring to that affect length-delimited values? Buffer overflows? Those aren't caused by values being length-delimited, they are caused by people deserializing variable-length values into fixed-length buffers without a check. That mistake can be made just as easily with a text format as with a binary format. In fact I've seen it much more often with text.
> JSON currently dominates large parts of that ecology.
JSON wins for one simple reason: it's easy for human developers to think about, because they can see what it looks like. This is very comforting. It's wasteful and full of pitfalls ("oops, my int64 silently lost a bunch of bits because JSON numbers are all floating point"), but comforting. Even I find it comforting.
Ironically, writing a full JSON parser from scratch is much more complicated that writing a full Protobuf parser. But developers are more comfortable with the parser being a black box than with the data format itself being a black box. ¯\_(ツ)_/¯
(Disclosure: I am the author of Protobuf v2 and Cap'n Proto. In addition to binary native formats, both have text-based alternate formats for which I wrote parsers and serializers, and I've also written a few JSON parsers in my time...)
> more comfortable with the parser being a black box
they're more comfortable with the parser being a black box but the format being relatively easy to parse compared to the parser being easy to understand but the format basically unreadable for a human.
The symbol parsing part of the human brain is what parses letters and numbers, as well as other abstract symbols. The division of symbols into letters, numbers, and others is fairly arbitrary. Most people would say “&”, but the modern name of that symbol is a smoothing over of the way it was recited when it was considered part of the alphabet and recited with it.
> I suspect this phenomenon can be observed in the case of relative popularity of JSON, TOML, YAML and Python plus the relative unpopularity of Lisp, Haskell, Rust, XML.
I suspect not: Lisp and Haskell have less use of non-alphanumeric characters than most more-popular general purpose languages, and not significantly more than Python; also, if this was the phenomenon in play, popularity would be TOML > YAML > JSON but in reality it's closer to the reverse.
I really don't think that's true when you talk about about someone using the latin alphabet, words in that alphabet compared to some other alphabet (e.g. {}():!) and "words" (or meanings) in those. Just as a crude example parsing "c = a - b", where equals and minus are one symbol each and have been taught for a while, is different from parsing "c := a << b" where ":=" and "<<" basically act as a separate meaning someone has to learn to understand. Similar to the difference of latin alphabet and say simplified Chinese.
> also, if this was the phenomenon in play, popularity would be TOML > YAML > JSON but in reality it's closer to the reverse.
There could be somewhat of an sigmoid response to the effect, decreased reaction if you go into either extreme compared to deviating from the average.
I'm not a linguist so it is my speculation, so don't take it too seriously :D
Especially since text can be arbitrarily long. From that perspective, length-delimited text (I've seen that before in a few file formats, and more notably HTTP) is probably the worst of both worlds.
Way worse than that. The interpretation of those different values of NaN is software specific. You can also have signalling NaN values - where the recipient can now have their number handling code trap in completely unexpected scenarios.
> Length-prefixed binary formats are almost trivial to parse
They definitely are not, as displayed by the fact that binary lengths are the root cause of a huge number of security flaws. JSON mostly avoids that.
> the most complex thing which might be required is endianness conversion
That is a gross simplification. When you look at the details of binary representations, things get complex, and you end up with corner cases.
Let's look at floating point numbers: with a binary format you can transmit NaN, Infinity, -infinity, and -0. You can also create two NaN numbers that do not have the same binary representation. You have to choose single or double precision (maybe a benefit, not always). Etc.
Similarly in JSON integers or arrays of integers are nothing special. It is mostly a benefit not to have to specify UInt8Array.
JSON is one of many competitors within an ecology of programming, including binary formats, and yet JSON currently dominates large parts of that ecology. So far a binary format mutation hasn't beaten JSON, which is telling since binary had the early advantage (well: binary definitely wins in parts of the ecology, just as JSON wins in other parts).