Okay, back to me:

>Actually, quite a lot in the past! Back in 2009 I did some XProc pipelining of messages. These pipelines were a bit like reactive streams, which were (mostly) agnostic of the contents. This allowed me to combine, dissect and route streams of data in an intuitive way.

Huh. So like this:

  |xmlstream|->|transformer|->|xmlstream|

Pretty slick. So the namespacing allowed you to add new tags, without worrying about tripping over the old ones? Cool, but the types of transforms you can do without knowing the internals of the XML you're transforming are fairly limited, and because JSON's objects don't mandate an app-wide wide meaning for a key - the closest thing JSON has to XML tags, you can just attach the new data to a new dict, and the problem solves itself. If you're merging objects, and each gives a different value for a key, than you can set up either an array or an object to hold both, or just send along both objects, wrapped in an array/object like before: in essence, by JSON'S semantics, each object is its own namespace.

>Reference, please? Even if I subscribe to one definition of 'wire protocol' on the internet (there are many), I don't think it creates a meaningful distinction between XML and JSON.

References, I can give. json.org, first paragraph:

  JSON (JavaScript Object Notation) is a lightweight data-interchange format. 

I apologize for being unclear: Data Interchange format is what I meant.

XML was not intended to be a generic data-interchange format: It, like HTML, SGML, and GML before it, were designed for DOCUMENT markup: Human-readable, structured, semantic DOCUMENTS. It has since been pressed into service as a data-interchange format, and it's a testament to how well it was designed that it works as well as it does for that, but its verbosity and general format and layout make it ill-suited to the purpose. JSON was designed for data interchange: I said wire protocol, as Data Interchange is often about sending data between applications on a network, which is what a wire protocol is for.

Hopefully some of that answers your question.

>Actually, it is, and it isn't. Yes, there are very few primitives (strings, booleans, numbers, arrays, objects), but this also causes important limitations. For example, it is rather cumbersome and unspecified to transfer binary data in a JSON document (base64 encoding). Another thing: how easy is it to parse a streaming JSON document in Javascript?

Is it specced to transfer binary data in XML? First I'd heard of it. Base64, uuencode, hex, or raw numbers, there are plenty of ways to encode binary data in JSON, and if you're using any system that has reserved characters (like CDATA in XML, if that's what you're thinking of), than you have to do this sort of encoding somehow. Besides, you could always send the json as a header, and have the app get the binary data from a different endpoint. Although you may want Base64 to avoid the roundtrips...

As for parsing streaming JSON, I don't know of there are any libraries for it, but the implementation should be very simple: Like XML, JSON is a tree, so parser state can be represented as a stack: You see a {, you're now in an object. A [, you're in an array. A , indicates adding a new value to the current array, or a new k/v pair to the current dict. What each character means is deterministic, given what came before, so you can construct JSON as the data comes in, and provide access to each value as it becomes ready. Although, given most JS implementations' multithreading limitations, all this really does is ensure that you don't have to have the entirety of the data in memory before you start parsing. Which is a good idea...

wrt the XProc pipelining: it's been some time, but I recall various possible transforming and matching steps, such as a conditional stream, transformers, reduction steps of multiple streams and others. This could then be combined with XPath, XQuery, XSLT and even SOAP requests. The problem of XProc was similar to the rest of the XML era: it required too much of the implementer to understand. Also, good programmer tooling, such as graph editors for the pipelines were missing. Perhaps this is slightly similar to functional programming and category theory nowadays. The ideas are sound, but it produces too much study and too little profit. Also, to properly work with category theory in programming, it would be very nice to have some graphical tools to view the transformations applied to your code.

Now for the flamebait-y parts. Yes, XML has loads of archaic SGML syntax bits, DTD built-in (hopeless for small parsers) and the attribute/sub-element divide has never been completely solved. But it can be argued that JSON had a similar fate: it literally descended as a subset of ECMAscript. This also explains the lack of separation between integers and floats.

But I agree, XML is definitely not the best data-interchange format, but neither is JSON. Some LISPy syntax would be my preference for data-interchange if it needs to be readable. But I'm trying to argue that this doesn't really matter. The XML era is mostly over, and I'd say we should try to learn from 'the good parts'.

I conflated XML with the XML Schema datatypes, and I shouldn't have, but it has been some time since I last seriously worked with XML. Also, we should consider the whole ecosystem, not just structure. XML Schema actually does spec binary data (http://www.datypic.com/sc/xsd/t-xsd_hexBinary.html).

With regard to round-trips: as a general rule it might not hold if you ask me. It implies an origin and even state! Maybe the sender cannot cache your binary data for a round-trip (memory, legal, latency, security constraints all play a role here). Personally, I like RESTful for simple systems, but for more involved architectures, message passing is much more scalable and easier to distribute.

> As for parsing streaming JSON, I don't know of there are any libraries for it, but the implementation should be very simple ...

Yes, a novice programmer should be able to write it in an hour. But the weird thing is, of all our libraries and all our frameworks (browser-side), none of them do streaming. Ok, I guess we should use websockets with JSON-encoded events for this, but still.

But hey, there are so many metrics with which one can evaluate a data-interchange format. (Recently we did a survey of binary data-interchange formats and found around 25 different criteria... and we were not really being thorough).

Okay. Cool.

I wasn't just trying to flame when I wrote: Talking to people who have different ideas and use different stacks is a good idea, and it teaches you things you didn't know before. And learning about stuff is why I use HN in the first place :-).

>The problem of XProc was similar to the rest of the XML era: it required too much of the implementer to understand. Also, good programmer tooling, such as graph editors for the pipelines were missing.

I will have to look up XProc now, because the things you've been saying sound really interesting, and it's clear I don't really get it.

>But I agree, XML is definitely not the best data-interchange format, but neither is JSON. Some LISPy syntax would be my preference for data-interchange if it needs to be readable.

I suppose. I love lisp considerably more than the next guy, but lisp structures technically only specify linked lists, which are O(n) for all data retrival. This is also how most implementations implement them. Also, JSON is similar, and trival to convert to that format

  ["like", {"key":"this"}]
  =>("like" (("key" . "this")))

Although you would idiomatically use symbols in many places where JSON uses strings.

>XML Schema actually does spec binary data

Once again my inexperience with xml shows. Thanks for letting me know.

>With regard to round-trips: as a general rule it might not hold if you ask me. It implies an origin and even state![...] Personally, I like RESTful for simple systems, but for more involved architectures, message passing is much more scalable and easier to distribute.

Firstly, I'm pretty sure REST implies a message-passing architecture. Correct me if I'm wrong.

Secondly, the round-trip idea sucks for a number of reasons, but I don't think it has to imply either. Let's say you have an endpoint at example.com/<userid>/lastmessage, which might give you the last message the user sent. If the user sent "just ate at Joes, #delicious," you might receive:

  {"message-type":"text", "message":"just ate at Joes, #delicious"}

But if the user sends an image, it's uploaded to the server, and you have to get it down. So you would instead get:

  {"message-type":"image", "message":"X57pqr32"}

And you would ask for example.com/<userid>/static/X57pqr32.

I don't know, but I think that would work.

>But the weird thing is, of all our libraries and all our frameworks (browser-side), none of them do streaming.

And I actually know why this is: Before the advent of Websockets, the only API was either XHR, or awful ideas (JSONP should chill the blood of any security expert). None of them supported reading incrementally, AFAIK, so there was no point. Now that Websockets are a thing, it shouldn't be long coming. Now all we need to do is build something to put JSONP in the ground...

>But hey, there are so many metrics with which one can evaluate a data-interchange format. (Recently we did a survey of binary data-interchange formats and found around 25 different criteria... and we were not really being thorough).

Indeed. By the way, did you look at Cap'n Proto and MessagePack? Neither are really on the fringe, but they look interesting, and they seem to have some decent support.

By the way, a good example of the multiple-trip RESTful API I described is XKCD's JSON API (http://xkcd.com/info.0.json)

Yeps, that's basically HATEOS :)

> Firstly, I'm pretty sure REST implies a message-passing architecture. Correct me if I'm wrong.

It absolutely is, but afaik (correct me if I'm wrong here) it implies an origin. It relies completely on addressable and available resources. It relies on exactly-once semantics (POST) and round-trips. Message passing for me is more like the actor model: ephemeral information, at most-once delivery, references to computers (actors), not data, and most important: the message is centric, not the end-point.

Perhaps I'm understanding all of this completely wrong, I'm honest here, but the actor model for me means 'message passing orientation' and RESTful to me means 'resource orientation'.

> And you would ask for example.com/<userid>/static/X57pqr32.

I implemented more or less the same scheme in a message centric application for crypto. Larger objects such as photos and videos were encrypted and placed in a central storage (later design phase included DHT implementation). The receiver could decrypt the message at a later time, whenever the photo was visible in the app/webpage. The central server, however, was non the wiser, as all data was encrypted and without semantic information. Here it is interesting to note that, even though we use references (URIs), the resource is not identifiable, except for its SHA hash signature. There was no sense in saying: https://kanta-messenger.com/photos/1234abcd since there is no knowledge of 'photo' or 'video'. However, there is still representable state transfer (REST) going on, without any of the semantics.

> . Now all we need to do is build something to put JSONP in the ground...

Agreed

> By the way, did you look at Cap'n Proto and MessagePack?

We had two phases (since it takes quite a lot of time to research each data-exchange protocol). In the first phase, we evaluated on a couple of core criteria: Language support (Scala, Java, Python), no long-standing github issues, more than one core commiter. We reduced that to three protocols: protobuf, Flatbuffers and Apache Avro. To most of our surprise, the last one won. Why? Various reasons, one of them being the possibility to do reflection and search within encoded messages for which the receiver does not have a schema. For example, you might want to create a router which only routes messages which contain a certain header. Another is archiving: since the schema is always included, it is possible to decode messages years after they have been stored somewhere. A third one is forward- and backward-compatibility. All of them were close wins (4 vs. 5 stars), but it brought us to Apache Avro. Looking back on that decision, it was a good one. Many within the company are happy with the choice.

>All of them were close wins (4 vs. 5 stars), but it brought us to Apache Avro. Looking back on that decision, it was a good one. Many within the company are happy with the choice.

Neat, I may check it out.

>It absolutely is, but afaik (correct me if I'm wrong here) it implies an origin. It relies completely on addressable and available resources. It relies on exactly-once semantics (POST) and round-trips. Message passing for me is more like the actor model: ephemeral information, at most-once delivery, references to computers (actors), not data, and most important: the message is centric, not the end-point.

I mean, that IS a valid way to think about it. I think about it like this:

when you're using a REST API, you are sending a message to an application. That application is defined in part by your endpoint: The server, and the path to the app. The rest of your message (params, method, remaining path) is your message. Some applications map the messages you send them onto a sort of virtual filesystem, which may or may not correspond to a real one. This appears in webservers, and many APIs. For these, messages you send primarily consist of paths. Others treat their messages more as procedure calls, and use more params. Both are messages, just as sure as

  cat /proc/sys/net/ipv4/ip_forward

and

  sysctl net.inet.ip.forwarding

even though one uses a filesystem model, and the other uses a command.

But your model of REST, while less linked to message passing, has much less cognitive load.

There's something wrong with me.

Actually, it's funny we're discussing message passing, because I've been working on an app that uses message passing between pre-emptive co-routines, and kinda-sorta unidirectional data flow heavily. Of course, at 2 coroutines per connection, it won't scale. Thankfully, it won't have to.

I hope.