News for hackers. Stuff that matters.
Re:RQ – Simple Job Queues for Python by elamje on Friday January 03, 2020 @03:14AM (#21944154)
I currently use RQ. Here is the logic that lead me to choosing it, then wishing I had just used Postgres.

I need a queue to handle long running jobs. I looked around the Python ecosystem (bc Flask app) and found RQ. So now I add code to run RQ. Then I add Redis to act as the Queue. Then I realized I needed to track jobs in the queue, so I put them into the DB to track their state. Now I effectively have a circular dependency between my app, Redis, and my Postgres DB. If Redis goes down, I’m not really sure what happens. If the DB goes down I’m not really sure what’s going on in Redis. This added undue complexity to my small app. Since I’m trying to keep it simple, I recently found that you can use Postgres as a pub/sub queue which would have completely solved my needs while making the app much easier to reason about. Using Postgres will have plenty of room to grow and buy you time to figure out a more durable solution.

Re:RQ – Simple Job Queues for Python by elcomet on Friday January 03, 2020 @10:49AM (#21946898)
Can't you track state in a "stateless" way ? For example having the job write it's progress in a file, so that if your app goes down, you can just re-read the file and know the state of your job?
Re:RQ – Simple Job Queues for Python by rcfox on Friday January 03, 2020 @11:04AM (#21947040)
Relying on the filesystem has its own drawbacks. If you scale horizontally, the local filesystem might not have the file you created because it's running on a different server. Mounting storage across multiple servers will introduce bottlenecks, and now you have to worry about the storage going down. If you're running within a Docker container, you need to make sure the file doesn't live inside the container or it won't be persisted.
Re:RQ – Simple Job Queues for Python by samsbox on Friday January 03, 2020 @06:58AM (#21945163)
May I ask, what (python) solution you ended up going with?
Re:RQ – Simple Job Queues for Python by elamje on Friday January 03, 2020 @08:26AM (#21945616)
I haven’t implemented the Postgres solution yet. It’s on my todo, but I will likely implement it myself, i.e. without a package.