Re:Diagnosing Memory “Leaks” in Python
by polpo on Tuesday September 08, 2015 @12:05AM
(#10184073)
We solved the Celery + long running + code reloading problem by having each push of new code be associated with a new Celery queue. On push of new code, start the new queue, and SIGTERM the old one, which will wait for any long-running tasks to finish before exiting.
[ Parent
| on HN ]
Re:Diagnosing Memory “Leaks” in Python
by ThePhysicist on Tuesday September 08, 2015 @04:18AM
(#10184554)
Hm that's an interesting idea, thanks for sharing this! How exactly do you perform the restart of the Celery service and where are your queues configured? I guess you don't specify them in /etc/defaults/celeryd?
Correct, we didn't use /etc/defaults/celeryd, or use a standard init/systemd/upstart/etc-based way of starting celeryd. Instead we used daemontools to add and remove celeryd processes with their own configuration. It took a bit of work, but ultimately let us do code updates with zero downtime.