News for hackers. Stuff that matters.
Re:The growth of command-line options, 1979-Present by kstenerud on Tuesday March 03, 2020 @08:51AM (#22474397)
This is another reason why I'm developing an ad-hoc data representation with strong typing. JSON was almost right, but failed because there are too many fundamental data types that cannot be represented in a portable way.

A general purpose ad-hoc data format with all the fundamental types, that doesn't require endless text parsing but can be converted to a 1:1 text format for human viewing, would be helpful in so many situations, including serialization for piping between unix tools.

https://github.com/kstenerud/concise-encoding#example

Re:The growth of command-line options, 1979-Present by tasogare on Tuesday March 03, 2020 @09:49AM (#22474838)
The comparison table doesn't seem fair, especially on the XML column. In particular, it is stated that XML doesn't support decimal and float, whereas XSD in fact defines 19 primitive data types including decimal and float. List are also supported implicitly because child nodes of an element are ordered. Moreover, the claim that XML doesn't support metadata is devious because any file can precisely be extended with any additional metadata (possibly in their own namespace), without client breaking if done well.

XML strength also lies in its extensibility, and its large number of compliant implementations, which goes way way beyond a single uncompleted implementation by a single author.

Re:The growth of command-line options, 1979-Present by int_19h on Tuesday March 03, 2020 @10:40PM (#22481031)
The problem is that you need an XML schema (or at least inline annotations on elements) to designate some value as decimal or float - it's not encoded directly in the data. I think that's what OP really meant.
Re:The growth of command-line options, 1979-Present by carry_bit on Tuesday March 03, 2020 @09:24AM (#22474639)
If JSON was almost right, why not EDN? https://github.com/edn-format/edn
Re:The growth of command-line options, 1979-Present by kstenerud on Tuesday March 03, 2020 @10:05AM (#22474968)
EDN is better than JSON, but it's still text only, and not fully specified.
Re:The growth of command-line options, 1979-Present by nicoburns on Tuesday March 03, 2020 @10:07AM (#22474982)
Have you seen RON (Rust Object Notation) https://github.com/ron-rs/ron ?
Re:The growth of command-line options, 1979-Present by kstenerud on Tuesday March 03, 2020 @10:13AM (#22475027)
Yes, but it's too rust-specific. For example, specifying struct names ties the document to a particular implementation, and isn't helpful for a general purpose data representation. Also, it's text only.

RON is fine for rust, but not for other languages that might not have its feature set.

Re:The growth of command-line options, 1979-Present by throwaway3157 on Tuesday March 03, 2020 @09:16AM (#22474574)
Cool. When you evaluated all the alternatives, did you stash a pro/con list somewhere on a blog? I’d appreciate reading it
Re:The growth of command-line options, 1979-Present by kstenerud on Tuesday March 03, 2020 @09:41AM (#22474766)
I made a short list here [1]. I also kept a design document which explains the choices I made [2].

[1] https://github.com/kstenerud/concise-encoding#comparison-to-...

[2] https://github.com/kstenerud/concise-encoding/blob/master/de...

Re:The growth of command-line options, 1979-Present by kentonv on Wednesday March 04, 2020 @11:58AM (#22485419)
Hmm, I think your table is using a different definition of "zero-copy" than is currently common when discussing serialization formats.

In your comparison you claim Concise is zero-copy, but it does not appear to be so, at least in the sense that Cap'n Proto and FlatBuffers are. Concise appears to use variable-width integers, tag-value sequences, and 8-byte alignment, all of which make encoded messages unsuitable as in-memory data structures; it looks to me like you have to parse the message into some sort of AST before you could meaningfully operate on the content. "Zero-copy" in the Cap'n Proto / FlatBuffers sense means that there's no need to do any "parsing" because the whole message is efficiently traverseable as-is; e.g. you can mmap() in a very large message and then find and access any one value within it in O(1) time (or perhaps O(log n), if the data structure is nested). This generally requires that primitive values have fixed widths and fields have fixed offsets within their parent object.

Digging into your docs it looks like what you really mean by "zero copy" is that individual strings or byte sequences can be used directly without copying them out of the original message buffer. This is a fairly common property of any binary protocol; e.g. Protocol Buffers can do this (albeit without NUL-termination), but in your table you have indicated that it cannot. I would maybe call this "zero-copy strings" to be clearer.

On another note, in your table you have indicated that Cap'n Proto doesn't have a "String" type, but this is not true -- Cap'n Proto's "Text" type is a UTF-8 string (and is even NUL-terminated on the wire).

(I'm the author of Cap'n Proto.)

Re:The growth of command-line options, 1979-Present by hwbehrens on Tuesday March 03, 2020 @09:29AM (#22474688)
There is a comparison table on the linked GitHub page, with the following formats: Concise (the proposed format), XML, JSON, BSON, CBOR, Messagepack, Cap'n, Protobufs, Flatbuffers, Thrift, and ASN.1.

Although not technically a pro/con list, it highlights the evaluation criteria used to compare.

Re:The growth of command-line options, 1979-Present by pinkfoot on Tuesday March 03, 2020 @08:53AM (#22474410)
ASN.1 ?
Re:The growth of command-line options, 1979-Present by kstenerud on Tuesday March 03, 2020 @09:01AM (#22474454)
Too complicated, too many extensions, too bloated on the wire.
Re:The growth of command-line options, 1979-Present by C1sc0cat on Tuesday March 03, 2020 @09:45AM (#22474806)
What you didn't have fun decoding OSI protocols :-)

I was a bit of a wuss and had to use x.409 dumps, my boss could look at the raw data packets and debug problems.

Re:The growth of command-line options, 1979-Present by pinkfoot on Wednesday March 04, 2020 @12:48AM (#22481493)
PER is bloated?