Re:Nginx 0mq transport layer for communicating with upstream...
by technoweenie on Friday January 06, 2012 @08:38PM
(#3435871)
Mongrel2's ZMQ protocol was designed for lightweight messages against async handlers. Instead of a synchronous ZMQ REQ/REP pair that this nginx module uses, Mongrel2 used a combination PUSH/PULL sockets to send the request, and PUB/SUB to get a response.
Say you want to map /chat to a handler. Mongrel2 opens a PUSH socket, and a SUB socket connecting to one or more chat handlers. The handlers open a PULL and PUB socket. Mongrel2 will spread requests to /chat across the available handlers, and receive all published responses. It's a pretty smart way to handle multiple handlers receiving requests at the same time. It's been a while since I looked at Mongrel2, so it's very possible I have some of the details wrong...
Re:Nginx 0mq transport layer for communicating with upstream...
by alexgartrell on Friday January 06, 2012 @07:29PM
(#3435711)
I was one of the first committers on Mongrel2 (actually, second behind Zed) and I still am (though mostly as a historical accident, I haven't contributed anything of substance since SSL).
In my opinion, this is great. I think Mongrel2 is an excellent proof-of-concept, and if it provided the impetus for a much more widely used project like nginx to go down a similar path (though it's just a module right now), then everyone wins.
That said, this certainly isn't in nginx mainline, and it's very feature poor as compared to Mongrel2. One could not simply swap in nginx with this module for an existing Mongrel2 installation.
Re:Nginx 0mq transport layer for communicating with upstream...
by piotrSikora on Saturday January 07, 2012 @04:09AM
(#3436616)
> it's very feature poor as compared to Mongrel2. One could not simply swap in nginx with this module for an existing Mongrel2 installation.
That's because this is _transport_module_ and what Zed did with Mongrel2 is build a level 7 protocol on top of ZeroMQ messages.
Even if this would support PUSH/PULL + PUB/SUB sockets right now, you would still need an _upstream_module_ that understands Mongrel2's level 7 protocol.
Re:Nginx 0mq transport layer for communicating with upstream...
by igrigorik on Saturday January 07, 2012 @01:09AM
(#3436351)
Alex, do you know if Mongrel2 have any interesting production deployments? In terms of visibility, throughput, or any other metrics. Outside from the architecture discussions, I haven't see any mentions of it.. (and I'm a fan of the architecture).
Re:Nginx 0mq transport layer for communicating with upstream...
by zrail on Friday January 06, 2012 @06:07PM
(#3435489)
From what I understand, core Nginx decided that supporting ZeroMQ was a non-starter because of the threading model that ZeroMQ uses. I'm not sure if that's just not true anymore, or if the restrictions that this new module places on it (namely, exclusively REQ/REP pairs) make that not apply.
Re:Nginx 0mq transport layer for communicating with upstream...
by swah on Friday January 06, 2012 @06:20PM
(#3435533)
I'm not sure how well adopted mongrel2 has been. There ain't many commits by Zed these days and I don't see blog posts about using it. I don't know why though.
Re:Nginx 0mq transport layer for communicating with upstream...
by daltonmatos on Saturday January 07, 2012 @04:17PM
(#3438096)
I've been using mongrel2 pretty much since the beginning. I host my own websites with it. Although they are very low traffic I ran some performance tests using blitz.io and the results were impressive. Less than 1% conn errors with a 512MB linode and a simple Django app (no database).
Re:Nginx 0mq transport layer for communicating with upstream...
by andrewvc on Friday January 06, 2012 @07:35PM
(#3435721)
Cool stuff, but I'm wondering why they went with REP/REQ instead of ROUTER/DEALER. That would dramatically reduce the number of file descriptors needed (indeed, a single socket could handle any number of backends / requests).
Also, the fact that ZMQ is message, not stream, oriented makes it non-ideal here. You get the whole response in one big chunk. Across an internal network, this may not be a huge deal, but it's not a desirable property.
It's conceivable that a variant that broke responses up into discrete messages might be better, but this would have more complication in implementation, and I wonder how much it'd have to offer over straight HTTP.
Also, I wonder if the true ideal protocol for this is SPDY without SSL. It just seems like a no-brainer.
Re:Nginx 0mq transport layer for communicating with upstream...
by piotrSikora on Saturday January 07, 2012 @04:37AM
(#3436663)
> Also, the fact that ZMQ is message, not stream, oriented makes it non-ideal here. You get the whole response in one big chunk. Across an internal network, this may not be a huge deal, but it's not a desirable property.
ZMQ and ngx_zeromq support multipart messages, so this is only true if your backend sends it as one big chunk.
Re:Nginx 0mq transport layer for communicating with upstream...
by JoshTriplett on Saturday January 07, 2012 @12:52AM
(#3436320)
> Also, I wonder if the true ideal protocol for this is SPDY without SSL.
SPDY does not exist without SSL, nor will it.
Also, SSL does not add anywhere near enough overhead to matter for this kind of application, especially when you just leave the connection established. SPDY would work quite well as a protocol for this purpose.
Re:Nginx 0mq transport layer for communicating with upstream...
by re on Saturday January 07, 2012 @03:28AM
(#3436559)
> SPDY does not exist without SSL, nor will it.
Can you elaborate on this, or provide a link with more context? Almost all TLS/NPN references are missing from the current SPDY draft spec. (I've found webpages that repeat the SSL requirement assertion, but none that source it or explain why.)
Re:Nginx 0mq transport layer for communicating with upstream...
by JoshTriplett on Saturday January 07, 2012 @03:56AM
(#3436599)
The SPDY draft spec doesn't seem to include any of the encryption bits; I don't know why. I've read documents explaining the details before (as well as expressing very strong opinions about the (lack of) merit of a modern protocol supporting a non-encrypted mode), but I can't seem to find them at the moment.
As far as I know, SPDY's protocol always builds on an SSL session, not a raw socket connection. Opportunistic SPDY connections to HTTP servers rely entirely on SSL "next protocol negotiation" (NPN); a SPDY server will indicate that it supports SPDY to an HTTPS client via NPN. SPDY also uses SSL to avoid interference from proxies that would otherwise break attempts to use SPDY on what started out looking like an HTTP connection.
Re:Nginx 0mq transport layer for communicating with upstream...
by piotrSikora on Saturday January 07, 2012 @04:04AM
(#3436609)
> Cool stuff, but I'm wondering why they went with REP/REQ instead of ROUTER/DEALER.
Because this is just proof-of-concept hacked over 2 days (+ README file). It probably wouldn't be even released at this point, if I didn't promise it to @igrigorik.
I'll look into adding other socket types sooner or later.
Wasn't Mongrel2 created solely because NGINX didn't support ZeroMQ.
@ZedShaw, thoughts - comments?
Say you want to map /chat to a handler. Mongrel2 opens a PUSH socket, and a SUB socket connecting to one or more chat handlers. The handlers open a PULL and PUB socket. Mongrel2 will spread requests to /chat across the available handlers, and receive all published responses. It's a pretty smart way to handle multiple handlers receiving requests at the same time. It's been a while since I looked at Mongrel2, so it's very possible I have some of the details wrong...
In my opinion, this is great. I think Mongrel2 is an excellent proof-of-concept, and if it provided the impetus for a much more widely used project like nginx to go down a similar path (though it's just a module right now), then everyone wins.
That said, this certainly isn't in nginx mainline, and it's very feature poor as compared to Mongrel2. One could not simply swap in nginx with this module for an existing Mongrel2 installation.
That's because this is _transport_module_ and what Zed did with Mongrel2 is build a level 7 protocol on top of ZeroMQ messages.
Even if this would support PUSH/PULL + PUB/SUB sockets right now, you would still need an _upstream_module_ that understands Mongrel2's level 7 protocol.
- PUSH/PULL + PUB/SUB support for ngx_zeromq (transport module),
- ngx_mongrel2 (upstream module).
Should be ready sometime next week.
ZeroMQ I/O threads are still being used with REQ/REP, so this doesn't change anything in that matter.
Also, exclusive use of REQ/REP pair isn't restriction, but current limitation of the module ;)
https://github.com/zedshaw/mongrel2/commits/develop
I'm also blogging about mongrel2 lately: http://daltonmatos.wordpress.com
It has nothing in common with Mongrel2 other than the fact that both use ZeroMQ transport instead of TCP.
Also, the fact that ZMQ is message, not stream, oriented makes it non-ideal here. You get the whole response in one big chunk. Across an internal network, this may not be a huge deal, but it's not a desirable property.
It's conceivable that a variant that broke responses up into discrete messages might be better, but this would have more complication in implementation, and I wonder how much it'd have to offer over straight HTTP.
Also, I wonder if the true ideal protocol for this is SPDY without SSL. It just seems like a no-brainer.
ZMQ and ngx_zeromq support multipart messages, so this is only true if your backend sends it as one big chunk.
SPDY does not exist without SSL, nor will it.
Also, SSL does not add anywhere near enough overhead to matter for this kind of application, especially when you just leave the connection established. SPDY would work quite well as a protocol for this purpose.
Can you elaborate on this, or provide a link with more context? Almost all TLS/NPN references are missing from the current SPDY draft spec. (I've found webpages that repeat the SSL requirement assertion, but none that source it or explain why.)
As far as I know, SPDY's protocol always builds on an SSL session, not a raw socket connection. Opportunistic SPDY connections to HTTP servers rely entirely on SSL "next protocol negotiation" (NPN); a SPDY server will indicate that it supports SPDY to an HTTPS client via NPN. SPDY also uses SSL to avoid interference from proxies that would otherwise break attempts to use SPDY on what started out looking like an HTTP connection.
Because this is just proof-of-concept hacked over 2 days (+ README file). It probably wouldn't be even released at this point, if I didn't promise it to @igrigorik.
I'll look into adding other socket types sooner or later.
thanks!