News for hackers. Stuff that matters.
Re:Django Styleguide by true_religion on Wednesday January 11, 2023 @08:47AM (#34338349)
My choice for periodic jobs is cronjobs.
Re:Django Styleguide by VWWHFSfQ on Wednesday January 11, 2023 @09:04AM (#34338558)
Do you mean a cronjob that calls a Django manager command to do the work? Or invokes an API method? From my experience cronjobs have a lot of downsides as well. They're great for doing tasks local to the server the job is running on. Not so great for the kinds of tasks (periodic or transactional) that Celery/a real queue is designed for.
Re:Django Styleguide by true_religion on Wednesday January 11, 2023 @09:48AM (#34339016)
I use them for invoking django commands on the same server. I do use celery for transactional jobs though. It’s only periodic jobs that get called with cron. For the context, I do this on small web apps with less than 20 dedicated servers (real servers not VPS), so there is a “manager” server that does nothing but run periodic tasks, management interventions, and cleanup operations.
Re:Django Styleguide by danjac on Wednesday January 11, 2023 @11:11AM (#34340001)
Crons are fine for running Django commands that are just talking to the same services as your web views (databases, cache, email services etc). It should be fine to let them run in their dedicated VM/containers, using Ansible etc to keep the crontab configurations in the repo.

Celerybeat has the advantage of better visibility e.g. you can configure them in the Django admin and check when they are running. If you are not using Celery though and your needs are simple, it's easier to just use plain crons.

Re:Django Styleguide by andybak on Wednesday January 11, 2023 @09:00AM (#34338514)
Yeah. Or a simple cron wrapper like django-cron to get the best of both worlds.

For background tasks - you can just spawn a background process and keep a simple status table in the db so the main app can check if it's completed (assuming you even need that)

And for task queues that can handle the traffic most sites will need there's things like django-huey.

Re:Django Styleguide by VWWHFSfQ on Wednesday January 11, 2023 @09:08AM (#34338607)
> For background tasks - you can just spawn a background process and keep a simple status table in the db so the main app can check if it's completed (assuming you even need that)

I don't know if making my own bespoke queue system is a great idea. It seems simple enough, but it gets so much more complicated once you start seeing issues with it. Orphaned task processes sticking around on the server forever, concurrency control, error handling, etc. I'll pretty much always just use celery and not have to worry about it.

Re:Django Styleguide by andybak on Wednesday January 11, 2023 @09:31AM (#34338847)
I'd never argue for building your own - just for using something simpler than Celery

It's not so bad now as CI/CD, Docker etc have made complex deployments easier to handle. But back when I was wrestling with Django simply deploying Celery on a new host could easily waste an afternoon and all those dependencies made me very nervous about the overall complexity.

I still weigh carefully anything that adds another long-running process or non-Python dependency to my sites.