Re:Ask HN: How would you queue and process 10K+ long running...
by ecesena on Tuesday November 27, 2018 @02:51PM
(#18546043)
There are great comments on the technology itself, including SQS/PubSub/RabbitMQ, or Celery.
However if you need 1) management, i.e. easy way to look at what went wrong and retry 2) dependencies between jobs, you should look into something like Airflow (instead of building your own): https://airflow.apache.org
It's also a very good example of architecture, in case you decide it's not good for you and you really want to build your own.
However if you need 1) management, i.e. easy way to look at what went wrong and retry 2) dependencies between jobs, you should look into something like Airflow (instead of building your own): https://airflow.apache.org
It's also a very good example of architecture, in case you decide it's not good for you and you really want to build your own.