I’ve found celery to be extremely useful and reliable at handling > 1MM daily async tasks with non deterministic latencies. The programming interface melds beautifully with Flask.
For the record, back in the days, Instagram used Celery at large scale : http://lc0.github.io/blog/2013/05/01/celery-messaging-at-sca...
Anecdotal, but we use Celery in a large scale production environment without too many issues.
Celery has been pretty reliable when i have used it in the past. Admittedly I have newer done anything particularly fancy with it, just running some background tasks and sending out some emails.
Re:Celery 5.0
by heavenlyblue on Tuesday October 06, 2020 @09:31AM
(#24697454)
So it didn't solve any problems that other simpler libraries would not solve, as I have said above.
- By default, task IDs are just pickled Python objects - if you want to change the location of a function, it might break; - The whole process appears to be reloaded for each task - any heavy loading ends up being performed on each call; - I couldn't find any "ops" documentation: how does it interact with RabbitMQ? How are deferred tasks implemented? What happens if a node crashes?
Although the API is nice, the product itself seems ill-suited for actual reliable, production use — at that point, it's sometimes easier to just deploy your own minimal API using the serialization format you've chosen to adopt (JSON, Protobuf, ASN.1, … ;)