Re:Scaling with Queues
by noelwelsh on Tuesday September 10, 2013 @11:38AM
(#6360663)
The post claims that RabbitMQ is too slow, but the description of the problem is very confusing to me:
"Later, I ... ran a loader.io load test. It failed this model due to very low throughtput, we performed a load test on a small 1G DigitalOcean instance for both models. For us, the STOMP protocol and slow RabbitMQ STOMP adapter were performance bottlenecks. RabbitMQ was not as fast as Redis, so we decided to keep it and work on the second model."
So they're using Redis as a queue for RabbitMQ :-) Just use Kafka, I say:
"Kafka handles over 10 billion message writes per day with a sustained load over the highest traffic hour that averages 172,000 messages per second. ... the footprint for the Kafka pipeline is only 8 servers per data center" http://sites.computer.org/debull/A12june/pipeline.pdf
"Later, I ... ran a loader.io load test. It failed this model due to very low throughtput, we performed a load test on a small 1G DigitalOcean instance for both models. For us, the STOMP protocol and slow RabbitMQ STOMP adapter were performance bottlenecks. RabbitMQ was not as fast as Redis, so we decided to keep it and work on the second model."
So they're using Redis as a queue for RabbitMQ :-) Just use Kafka, I say:
"Kafka handles over 10 billion message writes per day with a sustained load over the highest traffic hour that averages 172,000 messages per second. ... the footprint for the Kafka pipeline is only 8 servers per data center" http://sites.computer.org/debull/A12june/pipeline.pdf
That oughta be enough.