News for hackers. Stuff that matters.

A comment on Eliminating Task Processing Outages by Replacing RabbitMQ with Apache Kafka. Parent comment

Re:Eliminating Task Processing Outages by Replacing RabbitMQ... by zbentley on Wednesday October 07, 2020 @11:51AM (#24709442)
With your countdown implementation, how did you solve the resolution issue? I.e. if I delay a task for 111 seconds, I can publish it to RabbitMQ with a queue with a per-queue TTL of 100 seconds, after which it'll fall out to a DLX. What happens with the remaining 11 seconds? Does the DLX deliver (somehow) to a series of increasingly shorter per-queue TTL queues?

I, too, would love an implementation of arbitrary-granularity delays without buffering (potentially huge, mem-wise) tasks in "unacknowledged" in a giant binary heap: that's expensive for consumers and terrifying for RabbitMQ stability when a broker has to e.g. re-"ready" millions of messages because a delay-buffer OOMed.

[ Parent | on HN ]
Re:Eliminating Task Processing Outages by Replacing RabbitMQ... by avinassh on Wednesday October 07, 2020 @12:22PM (#24709767)
One solution which we did was to use the official delay plugin provided by the Rabbit MQ. It’s working good, so far

https://www.rabbitmq.com/blog/2015/04/16/scheduling-messages...

[ Parent | on HN ]
Re:Eliminating Task Processing Outages by Replacing RabbitMQ... by hnrodey on Wednesday October 07, 2020 @04:49PM (#24712557)
Hopefully you're aware of some of the shortcomings of this plugin when operating in a multi-node cluster.

The plugin keeps a node-specific database of the messages that are to be delayed. If the node is unavailable or lost, so too are the messages the node was keeping for future publish.

Also, there is no visibility in to the messages that are awaiting delay. 10? 100? 5000? Your guess is as good as mine and you'll never be able to figure out what's in to-be-published pipeline.

Signed, someone who has dealt with these issues.

[ Parent | on HN ]
Re:Eliminating Task Processing Outages by Replacing RabbitMQ... by zbentley on Wednesday October 07, 2020 @01:33PM (#24710602)
Awesome! I was peripherally aware of that, I think, but had filed it as "not ready" for some reason. Will definitely give it another look, thanks!
[ Parent | on HN ]
Re:Eliminating Task Processing Outages by Replacing RabbitMQ... by hnrodey on Wednesday October 07, 2020 @04:50PM (#24712565)
Hopefully you're aware of some of the shortcomings of this plugin when operating in a multi-node cluster.

The plugin keeps a node-specific database of the messages that are to be delayed. If the node is unavailable or lost, so too are the messages the node was keeping for future publish.

Also, there is no visibility in to the messages that are awaiting delay. 10? 100? 5000? Your guess is as good as mine and you'll never be able to figure out what's in to-be-published pipeline.

Signed, someone who has dealt with these issues.

[ Parent | on HN ]