automatthew
It's easy to sabotage standups in this manner by asking questions of others that you know will cause extended discussions. Especially on topics that have previously been "settled" by fiat.
Now the town had a cat problem. They imported a large number of dogs, and soon the rats were gone. But now they had a dog problem.
The town imported a large number of lions...
https://github.com/npm/npm/commit/2389525a1df6d20e780cca5887...
Workers can be written in any language, which was another major design consideration. We have about a dozen types of worker right now; when we want to evaluate a new language, we can port just one. Alternatively, we can write any new worker types in an arbitrary language.
Thus we're not married to JRuby.
Redis PUB/SUB is not suited for the task queues we use, because any number of subscribers will receive the messages. We want to guarantee that only one worker will act upon each message.
We actually tested a few designs using m2 and 0mq during our R&D phase. 0mq's push/pull sockets provide the same "take" behavior as the Redis RPUSH/BLPOP, so we definitely could have used m2 as the HTTP end of a similar architecture to what we use now.
One of the considerations that led to the choice of Redis was the transparency of the queueing and dequeueing. The messages in the queue are easily inspected, and the Redis MONITOR command helps greatly in debugging.
More important from a design perspective was our desire to hide the HTTP specifics from the workers. m2 pushes a JSON or tnetstring representation of the HTTP request to its workers, but we want the task we send to the workers to be generalized, stripped of information that is only meaningful for HTTP. We also want to classify the task by resource type and requested action, which allows us to use multiple queues. Multiple queues allows us to implement workers on an ad hoc basis.
M2 could work here if our request classification only depended on the URL. But that is a limitation we are not willing to accept. Request headers can be very useful in dispatching, especially those related to content negotiation (Accept, Content-Type, etc.)
There is an interesting hybrid approach using mongrel2: write one or more m2 handlers that perform the same function as our node.js dispatchers. I.e. m2 sends the JSON-formatted request to an m2 handler that deserializes it, removes the HTTP dressing, classifies the request according to type and action, then queues a task in the appropriate queue. A worker takes the task, does its own little thing, and sends the result to an m2 handler that knows how to re-clothe the result as an HTTP response and deliver it to the m2 cluster.
Regarding the question about queue behavior across replicated Redises:
I do not know for certain, but I do certainly hope it is not possible for an item in a Redis list to be popped by more than one client, no matter how the replication is configured.
With our architecture, we could relieve at least some of the strain on the task/result messaging system by using a cluster of Redis servers for the task queues. Each queue server might have its own cadre of workers listening for tasks. The return trip (getting a result from a worker back to the HTTP front end) is a little trickier, because it matters which HTTP server is holding open the HTTP connection. You could use PUB/SUB (which I believe is how m2 currently does it), or each HTTP server could be popping results from its own result queue.
When using a single Redis server, the only hard limitation we have seen with using the BLPOP operation is the number of client connections Redis can keep open. In case it's not clear, the BLPOP is blocking for the client, not the server.