I’ve used RabbitMq to do long running jobs. Jobs that take hours and hours to complete. Occasionally even 1-2 days.

It did take some configuring to get it working. Between acking appropriately and the prefetch (qos perhaps? Can’t remember, don’t have it in front of me). We were able to make it work. It was pretty straightforward it never even crossed my mind that this isn’t a correct use case for RMQ.

(Used the Java client.)

The short answer is "yes" but the questions that you should be asking are: A) How long am I willing to block the queue for additional consumers, B) How committed am I to getting close to exactly-once processing, and C) how tolerant of consumer failure should I be? Depending on the answer to those three questions is what drives your queue architecture. Note that this has nothing to do with time spent processing messages or "long running jobs".

Assume that your producers will be able to spike and generate messages faster than your consumers can process them. This is normal! This is why you have a queue in the first place! If your jobs take 5 seconds or 5 hours, your strategy is influenced by the answers to those three questions. For example -- if you're willing to drop a message if a consumer gets power-cycled, then yeah, you'd immediately ack the request and put it back onto a dead letter queue if your consumer runs into an exception. Alternatively, if you're unwilling to block and you want to be very tolerant of consumer failure, you'd fan out your queues and have your consumers checking multiple queues in parallel. Etc etc etc, you get the drift.

Keep in mind also that this isn't specific to RabbitMQ! You'd want to answer the same questions if you were using SQS, or if you were using Kafka, or if you were using 0mq, or if you were using Redis queues, or if you were using pg queues.

Except sqs has a limit for unacked messages of 12 hours

If I were designing a system with tasks that took that long, I'd probably want to find a way to checkpoint along the way and break the task down into smaller steps. I'd hate to lose 11 hours and 59 seconds of work if the task fails near the end. You can then use multiple queues to pass the task along between steps.

Bingo. This is exactly a setup I have seen before as well. Message recieved and acked, process kicked off, job state tracked in redis or similar.

at which point, if you've got to use a DB to track status, really why bother with the queuing system?

When you’re dealing with billions of messages, i think queuing systems may be tuned more for it?

I’d like to hear why people chose Kafka over some RDBMS tables.

Performance and distributed nature.

Billions of jobs with hours long time to complete seems like something no one would have the resources for.

Just wait till a start convinces someone to give them money for a system that allows anyone, anywhere to queue up infinite loops XD

I'm going to be honest, I think some 80% of Kafka users are overengineering it.

Kafka has specific use cases but it seems a lot of people just go "ok use Kafka here" and wait for the load that rarely comes

In an online environment, having your system go down during those rare times will eventually cost you your business.

Sure, then kafka stays online and the rest doesn't because you forgot some details.

It's all a matter of how much throughput you need. A queuing system can handle, in the same hardware, orders of magnitude more than a traditional SQL database that writes rows to disk in a page-oriented fashion.

If your load is, say, a few hundred writes/second, stick with the database only, and it will be much simpler.

how does that help if you still have to have a DB tracking status? you still need the same order-of-magnitude of DB throughput

No, because you only need to read and write ids and maybe timestamps to your db, both of which are trivially indexed, rather than the whole blob of your message payload.

In many cases, the message payload is (or should be) an ID anyway. It's seldom desirable for the message payload to include a copy of an external source of truth, because it can cause sync issues. There are exceptions, of course.

I don’t think it should be an ID - these platforms are really made for creating distributed event-driven systems.

The idea is your task should just run off an ID, no point passing all that data around.

How do you get the data relevant to that ID?

If the answer is a call to a shared database, you might as well not have RabbitMQ.

You do not need a database. It is trivial (and correct) to create a ~'<x>-status' topic. In the forward arc you are reliably propagating job requests (acked). In the backflow the processing status of job is posted for anyone interested. You can even propagate retries, etc. It is an MQ and RabbitMQ shines in defining complex dispatch toplogies.

Yeah but they already have a database, so it's not like they're adding a database to the system. And (as the article says) the database already contains state, so it makes way more sense to remove RMQ and hold all the state in the database (like they did).

Because a queuing system offers a different thing than a (relational) database.

You can build a queuing system with a database, but you have to do that. Some of the features and constraints of the database might even make your life harder than it has to be.

Instead, view it like that: there is a need for a queuing system and a job system. Either or both can be implemented using a database for certain concersn, but it can also be a custom implementation. It's not a great idea to mix the two things unless the operational and infrastructure costs and complexity outweigh the benefits of a clear separation.

There are libraries that implement queueing on top of databases that require very little setup by the user. For example https://github.com/timgit/pg-boss

I’ve had a positive experience with Procrastinate.

https://procrastinate.readthedocs.io/en/stable/

Right. However, please don't forget that there are very often inherit limitations regarding scaling or availability and also that many fully fledged message queues come with a lot of perks like access management and administration / debugging tooling and interfaces.

I'm not saying that libraries like pg-boss and co. cannot sometimes replace a full queue implementation. But the tradeoffs need to be clear.

Queues are good for connecting separated systems (like 2 or more separate companies).

To me, the difference here is a job queue vs a message queue

I do something like this in some projects of mine.

RabbitMQ is completely happy to do this. On the other hand, literally every python client library would fall over and/or wedge forever periodically. Celery in particular was a flaming pile of garbage (at least circa 2018).

I wound up vendoring the best of the bunch (amqpstorm, fyi), and adding a bunch of additional logic to handle all the various corner cases. That, plus a bunch of careful thought about how the queues are structured, and I've been pushing millions of messages a day without issue.

I wasn't able to get a lot of my changes upstream due to the extent I had to alter the assumptions the library made about control flow.

Don't worry, Celery has always been and will always be a flaming pile of garbage

For what it's worth, I found rq to be much nicer than celery but only used it in a relatively constrained fashion.

rq doesn't interoperate with other languages though, so if you use it then your workers can only be written in Python. :(

> They should have used RabbitMQ just to kick off the job, immediately ACKing the request, and job tracking/completion handled inside... a database.

That's what I think too. Use the message queue for communication and potentially work distribution and the database for the rest like internal state.