joombar

I realise not the usual format for a story. But to Gawk... in this world of zero downtime deploys this is remarkable.

As the sun rises in the UK, this is half the business shut down during business hours.

"And they're right- a boy who enjoys t-shirts and sports is at a tremendous social advantage over one prefers dresses and ballet"

Perhaps in some very conservative societies.

Where I live I'm confident that a boy who enthusiastically engages in things which actually interest him is at a social advantage over boys who halfheartedly pretend to enjoy in whatever his parents think is normal.

Ballet, for sure. Maybe not dresses so much, at least not if he is doing the wearing, but having an interest in the clothes of his female peers certainly isn't any kind of social problem.

For small messages it will be slightly slower in terms of CPU time because of JS vs native DOM parser. But for small messages the time taken with modern JS engines should be less than a monitor refresh.

For small messages on fast networks there really isn't very much time to save, so for this worst case in terms of looking for somewhere to optimise I think it is best to aim for no significant performance impact. The best improvements are where the messages are very large, on slow networks, or where the server can write out a stream.

The XHR2 spec specifies "While the download is progressing, queue a task to fire a progress event named progress about every 50ms or for every byte received, whichever is least frequent"

Because of "whichever is the least frequent", for messages that take less than 50ms from first byte to last, Oboe.js will get only one callback from the browser and notify all callbacks from inside the same frame of Javascript execution. This is exactly what we want it to do since it avoids rendering happening between the callbacks which would be very bad for overall download time.

I'm in two minds if Oboe should be dropped in to replace JSON AJAX calls, because everything will be accessed from a slow network one day, or if it should be used only for the very large, streamed REST responses.

Two different user agent strings for one version of one browser? That can change at any moment based on a list the site authors don't control? There's no way that'd ever confuse things!

Just merged in support for reading any stream in Node:

https://github.com/jimhigson/oboe.js#reading-from-any-stream...

  oboe( fs.createReadStream( '/home/me/secretPlans.json' ) )
   .node('!.schemes.*', function(scheme){
      console.log('Aha! ' + scheme);
   });
   .node('!.plottings.*', function(deviousPlot){
      console.log('Hmmm! ' + deviousPlot);   
   })
   .done(function(){
      console.log("*twiddles mustache*");
   });

Well, you could have a JSON which was valid at the start and invalid at the end. It'd parse the first bit ok and only throw an error when it got to the invalid bit.

Nothing gets parsed more than once. SAX parsers already parse streams, they're just not used very much because they're a pain to program with.

No, although that's a nice idea and something I'd like to look into.

Do you know how to do that from node?

Here's the little test service I wrote to stream out some gzipped content:

https://github.com/jimhigson/oboe.js/blob/master/test/stream...

In meantime, made a bug report: https://code.google.com/p/chromium/issues/detail?id=309092

Quick experiment says: Gzip can be written out as a stream ok. Can't comment on Apache but Node does it fine.

Firefox's xhr fires progress events for gzipped content but not Chrome's. Looking to see if I can find a way round it.

It should make most calls faster. Exceptions are for small JSON files or on networks that are fast enough there is no streaming effect (the whole file arrives very quickly)

For most sites there'll be some users where it will make it faster (mobile, slow internet) and others that it'll be about the same. If the network is unreliable it should help as well because when the connection drops you don't lose what you already downloaded.

"makers" = just me :-)

it is built on top of a sax parser.

Liable to change but "An approach to i/o for rest clients which is neither batch nor stream; nor SAX nor DOM." I'm writing it now.

I suppose you could make a binary equivalent if you needed to. You'd need to make some kind of binary matching language, maybe like Erlang's binary matching.

Adding XML/XPATH support would be a natural extension.

There's only a node side so far as I needed to write for some component tests. It'd work with anything that writes out valid JSON.

The client side works in Node right now as well as in the browser but it is a bit browser-y. It is on my home office Agile board to make it a bit more node-y.

If you don't care about old browsers you could use the same connection to keep the state updated as you used to download the original state.

I built this for more-or-less standard downloading, only quicker. But, yeah, if you set a server up to feed it you could use it for lots of creative things.

I haven't looked at MXHR but here's roughly what Oboe does:

1 Create XHR, listen to XHR2 progress event. 2 Use Clarinet.js SAX parser, scoop up all events. 3 From SAX events, build up actual JSON and maintain path from root to actual node. 4 Match that path (+ some other stuff) against registered JSONPath specs. 5 Fire callbacks if they pass.

Hadn't seen that before. Interesting link, thanks.

Very similar except JSON/JSONPath instead of HTML/CSS. Oboe runs fine in Node but I want to make the code a bit more standards-y. Ie, using Node's EventEmitters instead of the little pubsub I made for the browser.

I agree. If you're doing something slow/asynchronous like aggregating several http resources it is worth it to write out as early as you can but keep server-side the same if you can generate the whole JSON quickly.

Hmmmm, yeah. Use cases doesn't take up much space, I'll put that before examples because it explains better what it is for.

There's a cool thing you might try where you can download the historic messages, then continue to stream live ones over the same http. If you don't care about old browsers (anything without xhr2 progress events) should work fine to download and stream on same connection.

I'll have to test to be sure you still get streaming. It depends how the browsers handle xhr2 events with regard to gzip'd http. I /think/ it'll be fine but I need to check to be sure. Eg, with gzip on you still get progressive html rendering.

Should be fixed now. Thanks for pointing out.

There's a test explicitly for not allowing this:

https://github.com/jimhigson/oboe.js/blob/master/test/specs/...

Parsing is built on top of Clarinet (see where the name comes from?)

https://github.com/dscape/clarinet

Unquoted JSON in docs is a mistake. I'll take a look now.

I suppose the example is a little artificial. It isn't really for using some of the JSON response while ignoring the rest (well, you can use it for that but it isn't the main use).

I got the idea for this project working on data vis. Not all of the data was visible and we wanted to display the first bit of data quicker without waiting for all of it to arrive. We could have just sent the visible bit but it was good to have some data ready in the off-screen section for when the user scrolled.

Before that I worked on a service where we were aggregating 6 or 7 services into a single JSON. Some of the services were quicker than others but because the AJAX lib we were using waited for the whole response they all had to go at the speed of the slowest component.

We could have done multiple requests but it was more elegant to serve a whole page's json in one call. Also, we cached the slow services so they were only sometimes slow.

No worries. This is my masters dissertation btw.

It will work with any JSON but It'll work faster if you write the JSON out a bit at a time.

Consider if you are writing data from a db: you can either collect all the rows together then write them out, or you can write them out one at a time as you get them. If you write one at a time you are in a sense streaming even if you're not using a streaming protocol.

On a slow network reading any http is like a stream because you can use the first bit to arrive without waiting for the rest. The faster the network the less the benefit but it shouldn't end up worse.

No changes required. It should accept any JSON resource.

Having said that, there's a much bigger improvement if you're progressively writing out the JSON rather than doing it in one big lump.