And the problem here wasn't JSON in the first place anyway, it was
> The “back” stack is unlimited, meaning we’re saving more and more state until we explode
And storing previous state in serialized string in current state, instead of just having it stored natively. You'd have same problem if you tried to serialize <format> into a field in <format>
Length-delimited formats don't need (exponentially increasing) escaping.
Yes it was. They stored state in a huge JSON dump which ended up working like a zip bomb by the way they had to escape a string. This alone resulted in an exponential increase in memory that quickly became a problem.
Think about it for a second: would they experience this issue if they stored an in-memory stack per user? They wouldn't.
Heck, it seems that they could live with their JSON choice if they replaced '/' with any other inescapable character and converted it back to '/' when reading.
There are plenty of problems to go around this approach and overall system design, but the weakest link is undoubtedly how JSON was used.
4f 6e 63 65 20 79 6f 75 20 73 74 6f 70 20 6c 65 61 72 6e 69 6e 67 2c 20 79 6f 75 20 73 74 61 72 74 20 64 79 69 6e 67 2e
Of course people can read binary formats. It is just that they read them more slowly, and it requires quite a bit of exposure/practice, so that people usually invest in tooling to simplify the process.
I suggest that when you see a comment like this, you ask yourself “if I change ‘human readable’ to ‘easily human readable’ would I need to offer my correction?”
And devs don't care about just config files put API requests and responses and any other kind of structured/semi-structured communication and data.
Database dumps say hello. :)
It lacks comments. Has no way to offer context. Cannot impose limitations, and normalizing needs additional, often poorly supported RFCs (JSON-LD or pointers).
It also is a poor non-human readable format, as it lacks crucial types (datetime, decimal) is rather verbose (just imagine the terabytes bandwidth wasted, globally on all these commas and curly braces) and cannot be parsed streaming.
2. I’m still not convinced it’s the file format’s job to encode every possible data type. We used to have deserialization layers for that, but for some reason, we don’t do that anymore. Nowadays we expect every file format to be self-describing, which I think is wasteful and limiting. But that’s a matter of opinion, obviously.
3. What do you mean by “cannot be parsed streaming”? Or to be perhaps more precise, what’s missing in existing streaming JSON parsers? I haven’t used any of them yet.
I only found jsoncons to work reliable but at the cost of speed compared to something fast like rapidjson. And even jsoncons might have the problem that large strings have to be fully in memory, i.e., if someone has the fun idea to store a 1 GB string inside a JSON, then you still have a potential out-of-memory event at your hand.
There are other attempts to fix this problem with the jsonl format but that format feels so ad-hoc that I'm not convinced yet. It stores one JSON object per line. Of course, this comes with its own problems such as each JSON object potentially still being unbounded and, now that lines are used to separate JSON objects, you cannot format those JSON objects in a human-readable way anymore. Also, the first time I encountered a file like this, it simply had a .json extension and I was wondering why every JSON parser failed to parse that file. Because it isn't valid JSON and should have a separate extension.
For cases where a single string is truly massive, I would lean towards defining a maximum size to prevent denial of service attacks.
The issue is that JSON ended up being used as a configuration format too, and comments are critical there.
JSON5 is the best answer I have found.
Decimal is there, isn’t it, as part of ‘number’, which lumps together fixed-size integers, IEEE floats and doubles, bigints and bigfloats, and leaves it to the parser to make the right guess as to how to interpret them (often assisted by having the programmer specify a target data type)?
Lumping all those together as ‘number’ makes for a much simpler grammar, but I think that’s not worth it for its negative effects on interoperability. How many json parsers can (easily) read a value of 1234567890123456890E0000000001 and write back the same string to a json file?
Apart from datetime, I’d add duration as a basic type, too, even though that has the problem of being imprecise. Users will want to specify durations such as “1m” or “1y” (is that 365 days? 365.2425 to correct for leap years? Does it depend on interpretation by the user? is it a lunar year of 354/355 days as in the Islamic calendar? Etc)
Definitely not an improvement over XML. Hopefully the next popular standard data format will learn from the past and take the best of previous attempts.
In the encode/decode library we use, the generated toString were accidentally so close to json, that we made them exactly so (mostly had to add quotes). Now we can use json tooling on top of efficient messaging.
JSON truly is the new XML, both in terms of advantages and disadvantages. I wish people would stop using it for everything and realise that using a "human-readable" format for data that is 99.999999% not going to be seen by a human is absolute insanity in terms of inefficiency.