News for hackers. Stuff that matters.
Re:How to Manage Apache Airflow with Systemd on Debian or Ub... by villasv on Thursday December 26, 2019 @07:05AM (#21883273)
I have found that using KillMode=mixed is very useful when running Airflow with Celery workers. It allows the system to shudown gracefully coupled with TimeoutStopSec, so the workers will stop receiving new jobs but will finish their current jobs before exiting which is nice for auto scaling or spot instances on AWS (coupled with EC2 lifecycle hooks).
Re:How to Manage Apache Airflow with Systemd on Debian or Ub... by 1996 on Thursday December 26, 2019 @09:45AM (#21884116)
Interesting. Why do you care about graceful shutdown?

Why not kill ASAP and restart?

Do you have problems with load booming up sometimes?

Re:How to Manage Apache Airflow with Systemd on Debian or Ub... by akramar on Thursday December 26, 2019 @12:59PM (#21885492)
In some cases we need to do an update-deploy-restart while a DAG is still running (not even the one being updated). Then several minutes or hours later the child processes raise a segfault and the jobs they were working on fail, requiring restarting any of those jobs. I imagine a graceful shutdown would allow the job to finish up and the DAG to continue with the remaining jobs.