News for hackers. Stuff that matters.
Re:Celery, non-blocking code and quest against coroutines by harel on Thursday June 09, 2016 @12:32PM (#11870458)
Put a hard time out on the worker and it will be killed and restarted if hanging for X period of time.
Re:Celery, non-blocking code and quest against coroutines by methodover on Thursday June 09, 2016 @01:10PM (#11870725)
Yeah, I've thought about it. I guess I've been holding out hope that I could figure out the underlying cause. What's wierd is, it always affects just one type of worker -- the workers that handle sending an email. And it happens on all workers simultaneously, on all servers, of that type. Just that type.

Meh. I should just put a harakiri timeout like you suggest.

Re:Celery, non-blocking code and quest against coroutines by jvoorhis on Thursday June 09, 2016 @12:55PM (#11870612)
In my experience, celery takes some hand holding to stay alive. I generally configure it to recycle workers after N tasks, in addition to a hard timeout on tasks. I've also found in my configuration, using RabbitMQ as the bus, workers can become non-responsive after receiving too large a message.