Re:PinLater: An asynchronous job execution system
by fintler on Wednesday July 09, 2014 @08:16PM
(#8012931)
Just for comparison, LinkedIn is doing about 2.5 million messages per second (on average per day) with Apache Kafka.
http://www.slideshare.net/ToddPalino/enterprise-kafka-kafka-...
Re:PinLater: An asynchronous job execution system
by hatred on Thursday July 10, 2014 @10:19AM
(#8015040)
Precisely, this is the same query I had when I glanced down the post. It will be great if the author or someone from the team respond to the various trade-offs and why they did not go ahead with Kafka ?
Re:PinLater: An asynchronous job execution system
by omrvp on Thursday July 10, 2014 @03:44PM
(#8016775)
(I am the author of the blog post). I posted a reply below earlier this morning that hopefully addresses your question.
Re:PinLater: An asynchronous job execution system
by bigtones on Wednesday July 09, 2014 @10:30PM
(#8013279)
Looks like they needed to have built-in support for time based scheduled execution of jobs at a specific time in the future. Rabbit MQ does not have that (that I'm aware of) without doing some horrible hacks around one queue per message.
Rabbit's advertised throughput doesn't reach 100,000/sec, but you should be able to shard your way around that. Or use Apache Kafka, which has the sharding built in.