Re:Three quick tips from two years with Celery
by phunge on Wednesday July 29, 2015 @02:53PM
(#9970778)
Celery works, that's been my experience. It's buggy, complicated, and we've had unresolved issues, but it more or less does the job.
The language for building workflows is pretty half-baked, and there have been times when I've wondered whether RQ or raw message passing would be better in my situation. But it's better than any of the alternatives that I've had direct experience with.
Re:Three quick tips from two years with Celery
by sandGorgon on Thursday July 30, 2015 @02:27PM
(#9976959)
have you looked at RQ - would love to get a comparitive opinion.
Re:Three quick tips from two years with Celery
by caio1982 on Wednesday July 29, 2015 @02:42PM
(#9970700)
I don't know how much of it is truth but I've definitely had problems with Celery and memory leaks in the past myself. We can add this up to the hearsay pile though.
Re:Three quick tips from two years with Celery
by gsharma on Wednesday July 29, 2015 @03:24PM
(#9970990)
What do you use as a broker? We have seen these issues too and use Redis as the broker.
Re:Three quick tips from two years with Celery
by mpdehaan2 on Wednesday July 29, 2015 @02:58PM
(#9970814)
My experience is that Redis slows down considerably with a large amount of data in the queue, and also consumes large amounts of RAM.
I'm not sure I'd look to something that explicitly uses Redis as an alternative unless I could be sure the queue would be reasonably small.
Which, if the jobs are not going to be overwhelmed with backed up jobs, maybe that's fine.
Also, maybe we were doing it wrong.
Is there any truth to this ?