Re:Red Engine: modern scheduling framework for Python applic...
by wmichelin on Sunday July 03, 2022 @03:37PM
(#31970347)
How does it handle state and restarts? What happens if a job is scheduled to run "before 10am", then the entire server restarts at 9:55am, will it try to run that same job again when it boots back up?
[ Parent
| on HN ]
Re:Red Engine: modern scheduling framework for Python applic...
by Closi on Sunday July 03, 2022 @04:51PM
(#31970909)
It will run it if the scheduler is called before 10am according to the docs - at runtime the conditions must be met.
Re:Red Engine: modern scheduling framework for Python applic...
by wmichelin on Monday July 04, 2022 @01:52AM
(#31973858)
Right but what if it already ran? Should the jobs be written such that they are tolerant to re-runs?
Re:Red Engine: modern scheduling framework for Python applic...
by falcor84 on Monday July 04, 2022 @04:19AM
(#31974742)
I think that regardless of the scheduling tool, it is very worthwhile putting in the effort to make your tasks idempotent. I've had tons of cases when I had to repeatedly rerun failed tasks "in anger", and was always grateful to know I made it safe to do so.
Re:Red Engine: modern scheduling framework for Python applic...
by angrais on Sunday July 03, 2022 @03:55PM
(#31970498)
If it took less than 5 minutes to boot then I can't see why it wouldn't work.
How would that work for other schedulers? Also, if a server reboots that's quite bad all round anyway. Hopefully you'd be notified directly.
Re:Red Engine: modern scheduling framework for Python applic...
by vore on Sunday July 03, 2022 @06:38PM
(#31971606)
I think on the contrary, it's important to build software that's resilient to random reboots: you can't control things like losing power or your CPU failing, so the less toil you have to manage around that the better.
Re:Red Engine: modern scheduling framework for Python applic...
by mplewis on Sunday July 03, 2022 @04:45PM
(#31970860)
Other schedulers have a durable database of attempted runs. This doesn't seem to have anything like that.
Re:Red Engine: modern scheduling framework for Python applic...
by lovelearning on Monday July 04, 2022 @01:12AM
(#31973694)
From its docs, I understand that it has a task status logging layer[1] which can persist to SQL/Mongo (using another framework called Red Bird[2]). I haven't tried them out though.
[1]: https://red-engine.readthedocs.io/en/stable/tutorial/basic.h...
Re:Red Engine: modern scheduling framework for Python applic...
by dr_kiszonka on Sunday July 03, 2022 @05:07PM
(#31971034)
Would you have a recommendation for an easy to use Python scheduler with such a feature for a personal project?
Re:Red Engine: modern scheduling framework for Python applic...
by lobocinza on Sunday July 03, 2022 @09:07PM
(#31972464)
I think you can create ephemeral timers with Systemd if you're on Linux.
Re:Red Engine: modern scheduling framework for Python applic...
by staticautomatic on Sunday July 03, 2022 @07:28PM
(#31971902)
APScheduler
Re:Red Engine: modern scheduling framework for Python applic...
by dr_kiszonka on Sunday July 03, 2022 @08:23PM
(#31972207)
It looks pretty good. Thanks!
Re:Red Engine: modern scheduling framework for Python applic...
by MonkeyMalarky on Sunday July 03, 2022 @07:06PM
(#31971772)
Seconding this. I would love a middle tier job scheduler / manager for Python that has persistence. I feel like there's a missing middle between cron+scripts and the enterprise grade tooling built for ETL tasks.
Re:Red Engine: modern scheduling framework for Python applic...
by beci on Tuesday July 05, 2022 @03:58AM
(#31985582)
https://www.prefect.io/ is also pretty easy.
Re:Red Engine: modern scheduling framework for Python applic...
by tekromancr on Sunday July 03, 2022 @10:14PM
(#31972817)
Celery Beat is pretty good