As I worked on it though I realized it might be of general interested to people. Thus the example in the README of piping JSON through multiple tools without generating trivial changes that mess up diffs.

Most of the decisions I made were clear: no insignificant whitespace, object keys must be ordered, etc.

There are two things I'm still not sure about:

+ Son doesn't provide escape sequences for any Unicode character that JSON allows to be written unescaped. This includes U+007f (ASCII "delete"). Will that cause a problem for many programs? All the other ASCII control characters are required to be escaped by JSON, U+007f is the only one left out.

+ Son doesn't allow trailing zeros in fractions. This means you can't serialize `1.0`, you have to serialize it as `1`.

I was confident in the decision to take out scientific notation (it would be cool if JSON parsers actually treated numbers as being in scientific notation and tracked significant digits, but they don't so I feel like that ship has sailed). Trailing zeros are different though because some JSON generators do use them to distinguish integers from fractions. The problem is that many parsers don't care about them, so you end up in a situation where parsers are tossing out information about documents, meaning they can't serialize them faithfully again which is the whole point of Son.

Basically making JSON more machine friendly and stricter to parse undermines the original reason to use JSON. If efficiency and absolute correctness are important use something not designed to be forgiving to humans :)

I would not call a JSON human friendly due to its lack for comments.

[1] https://en.wikipedia.org/wiki/Bencode

1. Keep 1.0 as a special case to maintain the int/float distinction (it's a float; calling it a fraction is kinda-of-a-lie).

2. Refuse to handle floats at all, at which point people can pass [ <mantissa>, <exponent> ] for reals or [ <numerator>, <denominator> ] for rationals.

There is of course (3), "build a compliance suite and claim the parsers that toss out information are Incorrect", but that doesn't seem compatible with your postel-ish goals.

This happens not to be correct. By specification JSON numbers are just a series of characters, arranged in a certain way: https://tools.ietf.org/html/rfc7159#section-6

In practice though many JSON parsers will parse non-integer numbers to floats.

> 2. Refuse to handle floats at all, at which point people can pass [ <mantissa>, <exponent> ] for reals or [ <numerator>, <denominator> ] for rationals.

This is a really interesting idea. If you're writing something that's super important like medical software it would probably be worth considering. However, my goals are just to make minimal changes that improve JSON some while still keeping it fairly readable, so I think that means I should stick with allowing `123.456` or whatever. I'd like to try to keep an open mind on this though.

I think all the parsers I've used inflate 1 to an int and 1.0 to a float ... that or they don't, and I just thought they did.

Either way, many thanks for the correction.

[1] https://tools.ietf.org/html/rfc4506.html

Edit: Apparently used by NFS and ZFS.

But perhaps more interesting is the really great machine to machine communication of complex types research that went on in the 80's. Some was interesting (CORBA), some was really scary (ASN.1), and some very fast and minimalist (XDR).

I guess it's one of the things that fly under one's radar if one isn't actively using a protocol.

On Windows a similar niche is (mostly "was") filled by Microsoft RPC, an implementation of DCE RPC, which forms the underlying protocol for DCOM.

   sunrpc	111	tcp	SUN Remote Procedure Call	[Chuck_McManis]	[Chuck_McManis]						
   sunrpc	111	udp	SUN Remote Procedure Call	[Chuck_McManis]	[Chuck_McManis]

It describes a consistent way to hash an object without defining a new format.

How should the following be serialized?

{"öp":1, "op":0}

edit:

and unambiguous.

var x = "\u0307\u0323\u0073"

var y = "\u0073\u0323\u0307"

These are arguably the same grapheme when displayed on a screen, but they would be considered different keys in SON, and sorted in different spots. I'm actually unsure if most diffing systems would catch that they are different or not, and if so, how they would represent that difference to the end user (since just showing the grapheme would be useless as they are the same).

The specification is silent on normalization, so a SON pipe could output an object with those as keys (and still meet spec), but not be accepted by a conforming SON parser (that did unicode normalization on incoming keys). So really, the specification should say which of the canonical normalized forms to use, or that keys should specifically may not be normalized/modified in any way. This works fine if you're running something like JavaScript -> JavaScript -> JavaScript, but would fail if a (currently conforming) system in the middle defaults to normalized unicode representations. In other words, would a pipeline like Firefox -> Swift -> Python -> Swift -> Nodejs -> Swift -> Firefox, even if they had conforming SON implementations (of the current specification)?

I don't think RFC 7159 actually mentions Unicode normalization at all. If it was in there I think it would be mentioned in the section on string comparisons, but that just talks about going unescaped codepoint by unescaped codepoint: https://tools.ietf.org/html/rfc7159#section-8.3

EDIT: Does Unicode normalization change over time? If so we definitely have to leave it out and just go codepoint by codepoint, because we don't want the validity of a Son document to change.

Right now the reference implementation sorts by comparing codepoints one-on-by one. When it reaches a codepoint that's unequal or nonexistant it orders the string with the lesser or nonexistant codepoint first.

So in your example, after serializing from JSON and deserializing to Son we get `{"op":0,"öp":1}`.

Two remaining questions:

1. What's the most unambiguous way to describe this process?

2. Right now the comparison is on unescaped strings. Should it be on escaped strings instead?

There's an issue open to discuss this here: https://github.com/seagreen/Son/issues/1

EDIT: I am curious what Unicode recommends for language-aware sorting, even though we're not going to use it. Is this the right place to look? http://unicode.org/reports/tr10/

EDIT2: RFC 7159 has language about equality that's relevant to comparisons. I'm confident now we're on the right track: https://tools.ietf.org/html/rfc7159#section-8.3

Isn't sorting of Unicode characters defined under Unicode? I think the correct answer for "how do I sort my unicode strings" is "defer to unicode".

That may be quite expensive in the end, depending on object size and content.

NTFS, which has to build a b-tree of filenames in a permanently stable sort order, solved this by writing the collation table to the disk when it's formatted, and never changing it again. Which means the Windows shell still has to re-sort the filenames because the currently "correct" collation may be different.

Or are there problems with that as well that I'm not considering (beyond it being less human readable)?

Happily sorting is only used to order members within Son objects. Individual strings are never sorted, so there isn't a worry about sorted strings coming out messy.

EDIT: Not meaning anything is a plus! But I see your point about being un-googleable.

There are some other edgecases that you aren't considering (or that I missed)

Min and Max integer values - JavaScript has some pretty tight limits here

Keys should be in lexicographic order — you might want to be more specific. Is there a permitted subset of Unicode? Which normalization rule must be used?

Son seems like a natural place to put restrictions on them, but I'm not sure if there's way to do it that (A) still makes for a clean spec and (B) still allows everything floats can encode to to be used.

Also, no top level lists!

More info at http://stackoverflow.com/questions/16289894/is-json-hijackin...

> Piping JSON through multiple programs creates lots of trivial changes, which makes it hard to do things like take meaningful diffs.

But you can always put JSON through a pretty-printer which puts values into canonical form before diffing. I wouldn't bother turning it into a formal specification. And you mention 'No insignificant whitespace' in the README, so it's not like your format makes line-by-line diffs any clearer.

It's as if IT hasn't had enough solutions in search of a problem...

Too late!

> But you can always put JSON through a pretty-printer which puts values into canonical form before diffing.

Son is a starting point for building such a pretty printer. It takes care of messy details like eliminating the redundancies in string and number encoding, so all you have to do to specify the pretty-printer format is say where you want your newlines and how much to indent by.

I don't understand: such pretty printers already exist (e.g. jq, which does a whole lot more [0]). If you're transmitting JSON and want to diff two documents, just pipe them into jq or another pretty-printer with key ordering, then use one of many existing line-by-line diff tools.

[0]: github.com/stedolan/jq

  $ echo "1." | jq .
  1

It might be simpler to change jq and/or other pretty printers (though might be implementation or design reasons not to).

But the other advantage of a new project like SON is positioning not engineering, in our heads, not in the code: tool for the job. If so, the page should spend more time on what that job is (for when people search for that job).

And, as the jq example shows, tools not designed for this job mightn't do it exactly right.

EDIT I was reading offline, so didn't see the sibling reply.

Unfortunately, jq doesn't provide flags to control scientific/non-scientific notation or which characters are escaped, meaning if you want very tight control over the JSON generated it's not a full option.

(Consider that the motivating example in the Son README isn't all you might want to use Son for. For instance some people need consistent hashing of serialized JSON documents).

Side note: I respect your positivity and steadfastness in the face of a skeptical HN response.

The exponent syntax in JSON is fun because it allows numerous representations of 0 and 1 using e.g. 0e1 and 7e0

Definitely the toughest call when making a JSON subset like this.

I do like the comment about having the S-expression format for data interchange defined since 1997.

[1] http://wiki.laptop.org/go/Canonical_JSON

[2] https://datatracker.ietf.org/doc/draft-staykov-hu-json-canon...

It doesn't appear that Canonical JSON does anything about redundant escape sequences in JSON. I'm still looking into it to be sure. This is a big part of the motivation behind the Son spec, and represents about half the EBNF: https://github.com/seagreen/Son/blob/master/son.ebnf#L19

For anyone else who knows of more JSON subsets, if you report them here: https://housejeffries.com/page/7 it would be really appreciated. I definitely don't want to do duplicate work.

How should the following be serialized (if at all)?

{"a":0,"\u0073\u0323\u0307":1,"\u1E69":2,"Z":3}

Note that I'm using the \u forms even though they aren't allowed via the specification because, HN doesn't show unicode combining properly sometimes, and so that's it's clear to see that it's a different byte pattern.

`{"Z":3,"a":0,"\u0073\u0323\u0307":1,"\u1E69":2}`

See this issue for the discussion: https://github.com/seagreen/Son/issues/1

My current understanding is that RFC 7159 doesn't require Unicode normalization to be performed, if it does we're in trouble: https://tools.ietf.org/html/rfc7159

EDIT: s/alternative/competitor

I decided it was easier in my case to just output JSON.

That's the virtue of having a format which offers a canonical representation for data.

> No insignificant whitespace.

I.e., it's not human-readable. It also has no decent way to exchange binary data (there is no byte-sequence type: one must either use Base64 or an array of integers, neither of which is space-efficient).

It'd be nice to have a format which is both human-readable and has a canonical representation. Fortunately, such a thing already exists (and I've even linked to it once already today), and has since 1997: http://people.csail.mit.edu/rivest/Sexp.txt

Here's a JSON example (from http://json.org/example.html):

    {
        "glossary": {
            "title": "example glossary",
        "GlossDiv": {
                "title": "S",
          "GlossList": {
                    "GlossEntry": {
                        "ID": "SGML",
              "SortAs": "SGML",
              "GlossTerm": "Standard Generalized Markup Language",
              "Acronym": "SGML",
              "Abbrev": "ISO 8879:1986",
              "GlossDef": {
                            "para": "A meta-markup language, used to create markup languages such as DocBook.",
                "GlossSeeAlso": ["GML", "XML"]
                        },
              "GlossSee": "markup"
                    }
                }
            }
        }
    }

    {"glossary":{"GlossDiv":{"GlossList":{"GlossEntry":{"Abbrev":"ISO 8879:1986","Acronym":"SGML","ID":"SGML","GlossDef":{"GlossSeeAlso":["GML","XML"],"para":"A meta-markup language, used to create markup languages such as DocBook."},"GlossSee":"markup","GlossTerm":"Standard Generalized Markup Language","SortAs":"SGML"}},"title":"S"},"title":"example glossary"}}

    (glossary "example glossary"
              (div S
                   (entry SGML
                          (sort SGML)
                          (term "Standard Generalized Markup Language")
                          (acronym SGML)
                          (abbrev "ISO 8879:1986")
                          (def "A meta-markup language, used to create markup languages such as DocBook."
                               (see-also GML XML))
                          (see markup))))

    (8:glossary16:example glossary(3:div1:S(5:entry4:SGML(4:sort4:SGML)(4:term36:Standard Generalized Markup Language)(7:acronym4:SGML)(6:abbrev13:ISO 8879:1986)(3:def72:A meta-markup language, used to create markup languages such as DocBook.(8:see-also3:GML3:XML))(3:see6:markup))))

    {KDg6Z2xvc3NhcnkxNjpleGFtcGxlIGdsb3NzYXJ5KDM6ZGl2MTpTKDU6ZW50cnk0OlNHTUwoNDpz
    b3J0NDpTR01MKSg0OnRlcm0zNjpTdGFuZGFyZCBHZW5lcmFsaXplZCBNYXJrdXAgTGFuZ3VhZ2Up
    KDc6YWNyb255bTQ6U0dNTCkoNjphYmJyZXYxMzpJU08gODg3OToxOTg2KSgzOmRlZjcyOkEgbWV0
    YS1tYXJrdXAgbGFuZ3VhZ2UsIHVzZWQgdG8gY3JlYXRlIG1hcmt1cCBsYW5ndWFnZXMgc3VjaCBh
    cyBEb2NCb29rLig4OnNlZS1hbHNvMzpHTUwzOlhNTCkpKDM6c2VlNjptYXJrdXApKSkp}