I don't think this behavior has changed significantly. The key issue is that they seem to have correctly identified that they wanted to prefetch a single task, but they didn't recognize that this setting is the count of un-ACK'ed tasks. If you ACK upon receipt (as most consumers do by default), then you're really prefetching two tasks - one that's being processed, and one that's waiting to be processed. If you ACK late, you get the behavior that TFA seems to want. I've seen this misconfiguration a number of times.

> RabbitMQ isn’t a good fit for long running tasks

yeah I've seen 3 different workplaces run into this exact issue, usually when they started off with a default Django-Celery-Redis approach

all of those cases were actually easily fixed with Postgres SELECT FOR UPDATE as a job queue

I'll add another to the anecdata. We saw this issue with RabbitMQ. We replaced it with SQS at the time but we're currently rebuilding it all on SELECT FOR UPDATE.

Our problem was that when a consumer hung on a poison pilled message, the prefetched messages would not be released. We fixed the hanging, but hit a similar issue, and then we fixed that, etc.

We moved to SQS for other reasons (the primary being that we sometimes saturated a single erlang process per rabbit queue), but moving to the SQS visibility timeout model has in general been easier to reason about and has been a better operations experience.

However, we've found that all the jobs are in postgres anyway, and being able to index into our job queue and remove jobs is really useful. We started storing job metadata (including "don't process this job") in postgres and checking it at the start of all our queue workers and we've decided that our lives would be simpler if it was all in postgres.

It's still an experiment on our part, but we've seen a lot of strong stories around it and think it's worth trying out.

The linked docs in the article also suffer from the “wall of text” style that almost all software docs suffer from. If that’s what they had to go by to work out this behaviour, I wouldn’t be surprised they missed it. Most of the time when I look at docs, none of them are laid out to build understanding. It’s just a description of the system.

What exactly do you mean by "take a job in advance"? I have certainly set the prefetch limit to 1 on my queues, which I believe prevents them from taking a job while they are running one. One of our production applications runs 4+ hour jobs in a RabbitMQ queue without blocking and has done so for 5+ years.

You do want to make sure to set the VM high water mark below 50% of RAM as the GC phase can double the used memory. If high water mark is set too high, the box will thrash swap really badly during GC and hang entirely.