Why do we need Flask, Celery, and Redis? (2019)

Re:Why do we need Flask, Celery, and Redis? (2019) by mattbillenstein on Saturday April 18, 2020 @07:30AM (#22906975)

You still should use some sort of work queue - your application process may need to restart (deploys), or for a period of time, work could overflow the amount of available resources (bursts), so having some place to put the task before it goes onto processing is useful regardless of the underlying concurrency primitives of the language.

[ Parent | on HN ]

Re:Why do we need Flask, Celery, and Redis? (2019) by rollingbarreler on Saturday April 18, 2020 @09:44AM (#22907672)

Transactionality is enough for most systems. Your order either succeeds or fails, it never stays in an incomplete state. Queues are not a panacea, and introduce their own problems like obscuring problems by delaying them long enough to bring everything to a halt.

[ Parent | on HN ]

Re:Why do we need Flask, Celery, and Redis? (2019) by mattbillenstein on Saturday April 18, 2020 @08:17PM (#22912281)

Yes, you wouldn't do all work in the task queue - commonly, we make some change in the database which can happen pretty fast, and after that transaction commits, we might defer a task that sends a notification, email, whatever.

[ Parent | on HN ]

Re:Why do we need Flask, Celery, and Redis? (2019) by acjohnson55 on Saturday April 18, 2020 @11:24AM (#22908356)

It may not make sense to retain jobs across deployments. What if the contract of the job is changed by the code being deployed? Might be easier to keep it all in-process, letting queues drain in a graceful shutdown.

[ Parent | on HN ]

Re:Why do we need Flask, Celery, and Redis? (2019) by mattbillenstein on Saturday April 18, 2020 @08:20PM (#22912297)

I haven't found that to be the case typically -- you could always serialize some information into the task to check for things like this.

Also consider if the machine running that process just disappears and that process dies. Putting work into a task queue allows you to do it durably until it can be processed so that it's not lost in some typical "that machine/instance died" scenario.

[ Parent | on HN ]

Re:Why do we need Flask, Celery, and Redis? (2019) by sateesh on Saturday April 18, 2020 @02:41PM (#22910040)

This might not be viable all the time, what if you have a stream of tasks being put to the queue ? The alternative would be to ensure that any change in the job contracts are backward compatible, and if any change in contract would need to have a remediation/migration plan for handling pending tasks.

[ Parent | on HN ]

Re:Why do we need Flask, Celery, and Redis? (2019) by momokoko on Saturday April 18, 2020 @11:55AM (#22908550)

In the business, customer, end-user sense, in many situations, you are much better off to have a transaction fail with an error as opposed to finally successfully happening after 2 hours once a service disruption is cleared.

Every problem is not the same. Work queues introduce a magnitude or more of complexity to an http application. Sometimes that is very needed. Sometimes its overengineering. Sometimes it is a gray area.

[ Parent | on HN ]