I've been running four separate installs of Celery since 2014 or so processing about 10-20 million jobs a day, using reddit as a broker, and with about 100 worker instances per install. It just works, but I don't use Celery results or any esoteric features.

I used it for 10 years myself. Also learned the hard way not to use esoteric features.

I did have to build my own solutions for things like priority queues, but it was much clearer and testable.

Reddit :)

Posts to a subreddit as things in the queue, comments for results and feedback like progress percentages, up and down votes for priorities...

That's embarrassing! I'd like to blame autocorrect but I doubt it...

Did we just witness a Freudian slip? :)

Or overzealous phone auto correct. Redis used to auto correct to Reddit on an old android device of mine heh.

I can't stop giggling. ;)

I totally agree, I would not recommend using Celery. The last time I used Celery was in 2019 and it was a mess. The memory leak issue is still open https://github.com/celery/celery/issues/4843 to this day.

There has been a lot of progress on fixing memory leaks recently: https://www.paulsprogrammingnotes.com/2021/12/python-memory-...

IMO using python at all for this kinda setup with lots of workers is kinda crazy. As there is no usable threading, each worker need its own process. And as the article says, that's a couple of hundred mb of ram per worker. But in other languages, you could have hundreds of threads doing work with the same resources.

You can have async workers or gevent workers in python.

What would you recommend instead?

I'm also in the 'anything but Celery' camp:

https://dramatiq.io/motivation.html

If you're backing onto something like SQS or Rabbit (Redis is a bit trickier), it's not the worst thing to hand-roll either. Then you understand exactly what trade-offs you're making.

I had a good time with dramatiq+apecheduler, much more bare bones and less out of the box, but when something went wrong is was pretty easy to understand and work around.

Dramatiq is great.

For under 1000 messages per second, PostgreSQL is great at queues.

We have transitioned over to Dramatiq.

It has its own set of problems (mostly from starting at the set of "we will remove all of Celery features" and then going from there), in particular its logging policies are extremely frustrating and we have had to work around them. But the code is so simple in general that we get good compensation for that.

I personally would have liked to use dq instead but that's mostly an aesthetics thing.

Beanstalkd?

These comments say better than I can why I like it

https://news.ycombinator.com/item?id=27483167

https://news.ycombinator.com/item?id=24714198

Celery is not a broker daemon, it's an abstraction for them. You can use it with various brokers and some kv databases such as redis.

And yes, celery also supports beanstalk, so it's clearly not an alternative to it

http://docs.celeryproject.org/en/latest/getting-started/brok...

Currently exploring using arq, a successor to rq for my next project.

What have you switched to?

man... couldnt agree more.

Do you have any suggestions for alternatives?