That was a fantastic read. This is indeed old news in terms of technology choices, but it was great to hear how they rolled the product out.

I almost didn't read this article due to the "isn't this a bit old" comments here, but, am very glad I did.

I haven't used Erlang but I wonder if people will start blaming Erlang as "not stable" since Facebook chat is so unstable... the same way people said Ruby on Rails couldn't scale because Twitter was haven't problems scaling.

I believe its the Frontend implementation of the Chat that's unstable. Meebo has done facebook chat implementation pretty well.

I hope not, because it's hard to blame only Erlang.

First of all, they rolled their own epoll-based server for the whole chat system.

Then, instead of using a 'sane' comet/long-poll implementation based on JSONP (to avoid limitations of same-domain requests), it seems that they used a scheme where the chat nodes have a fake subdomain like 'http://N.subdomain.facebook.com/.... I don't know how efficient this is supposed to be, but I haven't seen that too often.

Of course, it's wholly possible their Erlang implementation of the chat system is the reason why it fails. Personally, I have never had much trouble with their chat, although I haven't used it much either.

The reason that they use N.subdomain.facebook.com is so they can avoid the default "2 http requests per domain" rule in older browsers.

I know that. What I don't get is why they chose that over JSONP, which solves the exact same problem by using <script> tags and a callback function.

Their solution seems to work well, anyway.

The jsonp method causes older browsers to use a ton of memory if you leave the browser window open for a long period of time. You'll see that there's a whole bunch of script tags in the dom that aren't cleaned up. Given the way people use facebook & how many old browsers still exist, this is probably the reason facebook decided against it.

jsonp also causes unsightly "loading" indicators in most browsers.

Plus w/ jsonp can't get the response from POST requests and you don't get notification that your page didn't load. So your web-stack needs to be able to handle sending any errors via jsonp as well.

In any case, they don't solve the same problem. Jsonp solves the problem of talking to another domain (PostMessage does the same thing). Channels solve the problem of the browser limiting the number of concurrent HTTP requests you can have open to one domain.

Given how easy it is to scale an Erlang app to provide incredible reliability in concurrent systems - as that is what the entire language is optimized for - this is pretty unlikely.

Is chat a 'concurrent system'? :/

Yeah.

Then again, the way Twitter overcame their scaling issues was to.. uh, well.. move off from Ruby for a portion of their stack.

I thought the same thing. :)

Isn't this a bit old and why does it go to posterous when it's taken from facebook.com?

It's very old, and the posterous owner thinks its fine to blatantly copy entire articles by merely providing "From: http://www.facebook.com/note.php?note_id=14218138919 as the only indication that it wasn't original.

Er... why didn't they just use XMPP? I haven't used facebook chat much, but the times that I have used it, it has proven to be a lot more unreliable compared to gTalk.

Having evaluated some XMPP solutions (over BOSH), one of the problems I have had with it was about the openness of the protocol. You need to tweak the server a whole lot if you want to close down what your users can do and from where they can do it; it becomes complex to synchronize all the data from the backend of the site: who ignores who, privacy settings, friend lists, etc.

Another problem that I had encountered was about BOSH. From what I had read in the RFCs, the BOSH session is based on random seed that you hash over and over in SHA (say 200 times). The server then just has to SHA your message to find the hash of the previous message. This, coupled with the basic session identifiers effectively makes it relatively complex to have the chat working over multiple windows or tabs.

There might be other reasons, but those are the obvious ones to me.

FWIW, Their Facebook chat seems to be constantly wrong. Unless it's just my account. "50 users online", then you click on it, and it changes its mind and says 7 users.

blogspam, actual URL is http://www.facebook.com/note.php?note_id=14218138919