News for hackers. Stuff that matters.
Re:Son: A minimal subset of JSON for machine-to-machine comm... by mehrdadn on Sunday December 10, 2017 @06:59AM (#15891059)
> but it does not do anything about one of the big omissions in JSON: it does not specify the range of integers or the precision of fractions

Could you give an example of a storage format that does do this and the problems that doing so avoids?

Re:Son: A minimal subset of JSON for machine-to-machine comm... by majewsky on Sunday December 10, 2017 @07:37AM (#15891144)
I'm not working with binary protocols that often, so I cannot point to an existing protocol. However, if I had to implement a serialization format that doesn't need to be human-readable, I would never use number literals. I would write numbers as one byte whose value means e.g. "the next 4 bytes are an uint32 in little-endian", followed by just these 4 bytes.

Then you also don't have to worry about whitespace, because there is none. The next field can follow directly afterwards, because the type header byte defines how long exactly the current field is. (For variable-length strings, use something like netstrings.)

The only problem that I can see with this is that Son wants to ensure deterministic encoding, but in my example, 0 could be encoded as an integer of any size or signedness. You could add a rule like "encode every integer as the smallest possible type", but that may have practical implications elsewhere.

I wouldn't be too worried about it, seeing how half-baked Son appears to be. For example:

> Object members must be sorted by ascending lexicographic order of their keys.

Yeah right, because there is only one lexicographic order in the world.

Re:Son: A minimal subset of JSON for machine-to-machine comm... by pegasuscollins on Sunday December 10, 2017 @09:39AM (#15891545)
> Yeah right, because there is only one lexicographic order in the world.

The specification tells you all strings (keys) must be UTF8. There is only one lexicographic ordering of UTF8 strings (RFC3629, page 2). This should probably be explicitly mentioned in the linked specification though.

Re:Son: A minimal subset of JSON for machine-to-machine comm... by majewsky on Sunday December 10, 2017 @03:20PM (#15893286)
> There is only one lexicographic ordering of UTF8 strings.

Then why is LC_COLLATE still a thing?

Also, you don't get to second-guess a spec. If it does not specify everything that it needs to, it deserves the "half-baked" label.

Re:Son: A minimal subset of JSON for machine-to-machine comm... by pegasuscollins on Sunday December 10, 2017 @09:52AM (#15891604)
> The only problem that I can see with this is that Son wants to ensure deterministic encoding, but in my example, 0 could be encoded as an integer of any size or signedness.

To be pedantic: Not really. Assuming your example uses twos complement (what else would it use) and you meant "int32" instead of "uint32", negative zero and positive zero would still be the same encoded value. Also since you said "[u]int32", there is only one size the integer can be: 32 bit. So, in your example, the literal value zero has one and only one unique binary representation, which is "0x00000000"

Re:Son: A minimal subset of JSON for machine-to-machine comm... by majewsky on Sunday December 10, 2017 @09:57AM (#15891631)
I implied that a useful serialization format would include integers of multiple size. When your field only needs a uint16, you don't want to pay the extra 6 bytes to encode as, say, uint64.
Re:Son: A minimal subset of JSON for machine-to-machine comm... by pegasuscollins on Sunday December 10, 2017 @10:01AM (#15891653)
You could either stipulate (as you suggested) to always use the smallest width that still fits a number and then prefix with a length byte but that would be wasteful. Ideally something like LEB128 would be used. At any rate, it's not an issue in terms of guaranteeing a bijective mapping.
Re:Son: A minimal subset of JSON for machine-to-machine comm... by majewsky on Monday December 11, 2017 @03:33AM (#15895830)
Nice, I didn't know about LEB128.
Re:Son: A minimal subset of JSON for machine-to-machine comm... by pegasuscollins on Sunday December 10, 2017 @08:57AM (#15891372)
Pretty much every binary format will encode integers using a fixed width or varlen scheme in "base 2".

This generally is done for two major reasons: First of all, such an encoding is significantly easier and cheaper to parse than a base10+ascii (human-readable) encoding.

I encourage you to write a parser that reads a fixed, 32-bit binary number (1) and another parser that reads a JSON-formatted number string into an internal variable in a classic language like Java or C++. You will immediately see the big difference in complexity. Make sure your parser can also deal with a message that contains more than just one number, i.e. the parser should be able to tell at which byte index an encoded number begins and ends. Even if you're using a language or library where this is hidden from you (e.g. by using parseInt or std::stod) the same work still happens behind the scenes.

The other reason is that for most numbers and fixed/varint encoding schemes, the "binary" representation will be much more compact. Storing the number "1000000" in base10+ascii (human readable) takes at least eight bytes. Storing the same number in a fixed 32-bit integer encoding takes four bytes. Using a varint encoding scheme might allow you to get down to three bytes.

(1) Ignoring stuff like byte order and representation of negative numbers; this is usually fixed in the protocol/format specification.