S-expressions might have caught on for this purpose, but they lacked a killer app (jquery, and later rails). Like JSON, they are easy to parse and easy to generate, but being more loosely-specified than JSON, it's less clear how to map a given S-expression to a native type. As a glue format, JSON feels just right, even if it's a bit picky sometimes (e.g. trailing commas).
For anyone who wants extra features like comments, YAML is the oldest popular format I am aware of that is a superset of JSON. For any new format to succeed, IMO it needs to sufficiently distinguish itself from both YAML and JSON, and not just support a feature set that happens to lie somewhere between the two.
I was doing mostly Perl and Javascript when it caught on, and to this day I have very mixed feelings: it's wasteful of space but still doesn't allow comments; its type system is basically a technical-debt generator; and for all that people still get it wrong pretty often. On the other hand, it's more or less human readable for simple data structures and it's more or less everywhere.
My hunch is that for a new format to take off it wouldn't so much need to not fall between YAML and JSON: it would need to be the default format of something with such super exponential growth that even us oldies would have to use it.
I believe it's hard to explain JSON's popularity and wide adoption without talking about javascript. With javascript, JSON was right from the start an ‘eval’ away from being parsed. The barrier to entry to adopt it simply was never there. Once you start to expose JSON APIs to clients, other servers also start to need to consume data from those servers. Rinse and repeat until you reach mass adoption.
They would not have been loosely-specified if they were specified, like JSON was :-) I mean this is taking things a bit backward. When tools use a data format based on S-exprs, they define more clearly what is or isn't valid (OCaml Dune, Guix, etc.)
E.g. XML has all that, and yet the average use of xml is much more fragile thsn the average use of json. Similarly asn.1 has all that, but does anyone actually like the fact the x509 certs use it? (To be clear, json would definitely not be appropriate for that case either).
- HORRIBLE schema format.
- Very few FOSS libraries, most of which are terrible. I used Lev Walkins asn1c[0], which I patched slightly and coupled with some semi-generic code that walks the generated data structures to emit JSON (for debugging purposes).
It was very handy to be able to use jq on the data.
Yes, but they all implement the JSON RFC slightly differently, or implement it in a totally non-compliant way.
However, we're coming from XML, which was better defined, and there are also binary "quasi standard" formats like protobuf, so those problems had already been solved.
Yet still "the world" has moved to JSON, which seems to indicate that those problems probably were not all that important for most use cases.
It's very easy to do JSON wrong, to not even be aware of doing it wrong, and do it wrong in a way that limits its interoperability (see: the default behaviour of the json library in Python). This is not theoretical either, just google for 'json nan github' to see the hundreds of production programs accidentally emitting JSON that cannot be ingested by RFC-compliant parsers.
And regarding integer ranges: Any user has restrictions on top of the generic format. Some fields have to be present for the application to work, some fields have to be a string, others an array. Some integer has to be between 0 and 100, some array has to have 5 elements. Given that 99% of integers in practice fit in a signed int32 there is little problem (97% are probably 0, 1% are 1, 0.5% are -1 and only the rest other values ...) If you are on the edge you have to know and work-around ...
The problem is that you are not guaranteed to know, as a JSON library user, that the library has not mangled the numbers it received on the wire prior to your application receiving it - so you don't know if you having .a set to 42 is the result of 42 being sent over the wire, or an implementation dropping bits that are outside the range of you library's support. The RFC does not mandate what a JSON implementation should do in case of numbers outside its' support.
> Given that 99% of integers in practice fit in a signed int32 there is little problem.
Until you hit that 1%, and you find this out the hard way, and you have no way of solving it because you don't control the emitting side. Again, this has happened in practice to me, when a Python library was emitting large numbers as numbers (as the RFC permits), while the receiving side silently casted to 32-bit floats, losing data (as the RFC permits). Both sides are right per the spec. Technically, everything worked as per spec. Practically, the product was broken.
99% of the time you might be okay. But the 1% of edge cases makes it that you can never rely on JSON, which makes it a bad interchange format if you care about reliability and safety. You _can_ make it work if you severly limit yourself and are deeply aware of all the possible issues that using JSON has (and there's a lot more). Or you can just pick some other standard that solves these basic things for you (eg. Protobuf).
Random side note: ECMAScript has no integer type, but only Number, which is a float, thus when dealing with such numbers in a JavaScript frontend you are in Problem Land anyways ... which again shows that boundaries have to be thought of, independently from the specification of the data exchange layer.
Had to look that up. Was under the impression the rfc only specified the syntax. But here it is
> This specification allows implementations to set limits on the range and precision of numbers accepted.
I don’t think the implication is that it must be done by silently loosing precision though. A loud error would fit that description just fine. So in this case I would blame the implementation not the spec.
- strongly typed
- ad-hoc or schema (your choice)
- no code generation step
- edit in text, send in binary
- supports ad-hoc data structures or schemas per your preference
- supports all common types natively (doesn't require special string encoding like base64 or such nonsense)
- supports comments, metadata, references (for recursive/cyclical data), custom types
- doesn't require an extra compilation step or special definition files
- Has parallel binary and textual forms so that you're not wasting CPU and bandwidth serializing/deserializing text. Everything stays in binary except in the rare cases where humans want to look or edit.
It's actually 2020-12, which is two versions after Draft 7 (they shifted from Draft n to YYYY-MM after Draft 7, and since then have had 2019-09 and 2020-12.)
And that's true of most languages, though there is some 2019-09 support. (It really doesn't help that there is also OpenAPI which baked in a variant—“extended subset”—of Draft 5 JSON Schema.)
https://github.com/lloeki/ruby-skyjam/blob/master/defs/skyja...
gives that:
https://github.com/lloeki/ruby-skyjam/blob/master/lib/skyjam...
I would certainly enjoy having a DSL to write descriptive code to validate using JSON schema, but it would be even better if the Ruby definitions could be generated and persisted in Ruby files using that DSL.
Also, storing things in basic hash/array types works, but having dedicated types is useful, so that one can ensure not shoving one kind of hash in place of another unrelated kind of hash.
As for types themselves in general, there's RBS and Sorbet. One could have type definition generation as well for even deeper static and runtime checks.
Existence of a schema definition file and checking responses against is signalling that I can trust an API vendor to be at least aware of the requirements for clients. (Whether they randomly change the schema definition or ignore it is a second question, but at least somebody once thought about formalising and it's not an complete adhoc dump of today's internal data representation)
"Amazon Ion is a richly-typed, self-describing, hierarchical data serialization format offering interchangeable binary and text representations. The text format (a superset of JSON) is easy to read and author, supporting rapid prototyping. The binary representation is efficient to store, transmit, and skip-scan parse. The rich type system provides unambiguous semantics for long-term preservation of data which can survive multiple generations of software evolution."
Apache Avro has support for parsing and utilizing schemas at runtime, even in C++.
For Apache Thrift you have things like thriftpy: https://thriftpy.readthedocs.io/en/latest/
I'm not aware of a type-safe mechanism for Flatbuffers or Protocol Buffers.
I never got anything out of xml validation either. It just lead to this ridiculously verbose garbage xml that was even less readable with all the namespace, namespace declarations, etc. Very tedious to write manually as well. Also lots of weirdness doing e.g. xpath or xsl transformations against that. The whole SOAP / web services bubble eventually imploded when people figured that you could send tiny json objects instead via REST.
I have many issues. People sending invalid json to my APIs is not one of them. You get a bad request if you do. End of story. Don't don that if you don't like bad requests. It's not a problem that justifies a lot of over engineering.
Json, yaml, toml, hocon, etc. are basically all just variants of attempting to send human editable blobs of information. Fine for apis where all of the requests are created by programs. Unfortunately people also abuse them for things like DSLs that are authored by humans. The real problem there is not using a strongly and statically typed language that simply does not allow illegal things. A schema is just a stop gap solution when you don't have that.
Kotlin is actually great for creating proper DSLs. You get IDE auto-completion and red squiggly lines when you do it wrong. I think Rust also has some nice syntactical constructs for creating DSLs. Typescript might also emerge as a language that is very suitable for that (with maybe a few more features borrowed from Kotlin). Using a non compiled language with weak typing kind of defeats the purpose. Hence the endless ruby but not quite ruby like DSLs for things like puppet.
There is no official JSON schema RFC, there is no official linking between json-schema.org and the JSON RFC. There is no push to make every language that supports JSON to also support a JSON schema system.
Finally, there isn't even a guarantee that all human-readable JSON document specifications can be expressed within a JSON schema in any sensible way.
JSON is something we can (almost) all agree on for dumping loosely specified, human readable representations of data structures. It lets users and client application developers be lazy and not have to learn a new library for our preferred data format. The lack of schemas or validation means we can often get away with wishy-washy, hand-written, plain English specifications, or specifications "by example". It's convenient.
For anything intended to be robust, for serious data interchange, use something else. Anything with support for machine-readable schemas, validation, and robust encoding rules. Binary is preferable but a JSON serialization is still a must-have escape hatch.
And here's a shocking idea... provide your clients with an SDK. It's pretty easy to build one around a multi-language data serialization framework and maintaining an SDK is still easier than maintaining a spec that your clients will probably implement incorrectly.