Re:PinLater: An asynchronous job execution system
by ritwikt on Wednesday July 09, 2014 @07:17PM
(#8012746)
Would be nice to know why RabbitMQ/ZeroMQ didn't fit the bill [scale/speed?] -- I can somewhat understand the not relying on SQS but then there is anyways a reliance on EC2 right ?
[ Parent
| on HN ]
Re:PinLater: An asynchronous job execution system
by bjt on Wednesday July 09, 2014 @07:42PM
(#8012839)
I thought the same thing about Rabbit. PinLater's ACK system feels like a questionable reinvention of AMQP. I'm probably missing something.
Rabbit's advertised throughput doesn't reach 100,000/sec, but you should be able to shard your way around that. Or use Apache Kafka, which has the sharding built in.
Re:PinLater: An asynchronous job execution system
by fintler on Wednesday July 09, 2014 @08:16PM
(#8012931)
Just for comparison, LinkedIn is doing about 2.5 million messages per second (on average per day) with Apache Kafka.
http://www.slideshare.net/ToddPalino/enterprise-kafka-kafka-...
Re:PinLater: An asynchronous job execution system
by hatred on Thursday July 10, 2014 @10:19AM
(#8015040)
Precisely, this is the same query I had when I glanced down the post. It will be great if the author or someone from the team respond to the various trade-offs and why they did not go ahead with Kafka ?
Re:PinLater: An asynchronous job execution system
by omrvp on Thursday July 10, 2014 @03:44PM
(#8016775)
(I am the author of the blog post). I posted a reply below earlier this morning that hopefully addresses your question.
Re:PinLater: An asynchronous job execution system
by bigtones on Wednesday July 09, 2014 @10:30PM
(#8013279)
Looks like they needed to have built-in support for time based scheduled execution of jobs at a specific time in the future. Rabbit MQ does not have that (that I'm aware of) without doing some horrible hacks around one queue per message.
Re:PinLater: An asynchronous job execution system
by kondro on Thursday July 10, 2014 @01:14AM
(#8013732)
ActiveMQ's had that for a couple of versions though. Another highly performant/available, atomic, consistent, durable and mature queueing server.
Re:PinLater: An asynchronous job execution system
by cyounkins on Thursday July 10, 2014 @11:02AM
(#8015185)
Correct. Celery's implementation dequeues but doesn't acknowledge the task, holding it until the timer is reached.
Re:PinLater: An asynchronous job execution system
by zackkitzmiller on Wednesday July 09, 2014 @07:41PM
(#8012836)
I'm curious here as well.