Re:Celery, non-blocking code and quest against coroutines
by harel on Thursday June 09, 2016 @12:32PM
(#11870458)
Put a hard time out on the worker and it will be killed and restarted if hanging for X period of time.
[ Parent
| on HN ]
Re:Celery, non-blocking code and quest against coroutines
by methodover on Thursday June 09, 2016 @01:10PM
(#11870725)
Yeah, I've thought about it. I guess I've been holding out hope that I could figure out the underlying cause. What's wierd is, it always affects just one type of worker -- the workers that handle sending an email. And it happens on all workers simultaneously, on all servers, of that type. Just that type.
Meh. I should just put a harakiri timeout like you suggest.
Re:Celery, non-blocking code and quest against coroutines
by jvoorhis on Thursday June 09, 2016 @12:55PM
(#11870612)
In my experience, celery takes some hand holding to stay alive. I generally configure it to recycle workers after N tasks, in addition to a hard timeout on tasks. I've also found in my configuration, using RabbitMQ as the bus, workers can become non-responsive after receiving too large a message.