Ask HN: How would you queue and process 10K+ long running jobs

Re:Ask HN: How would you queue and process 10K+ long running... by mtmail on Tuesday November 27, 2018 @09:09AM (#18542357)

Look for job scheduling software plus the name of your programming language of choice. Or background processing. https://sidekiq.org/ is one for Ruby for example, https://aws.amazon.com/sqs/ one that runs in the cloud. Those pages should give you more words to search for as "job scheduling" and "queue" gives too many non-software related results.

[ Parent | on HN ]

Re:Ask HN: How would you queue and process 10K+ long running... by wallflower on Tuesday November 27, 2018 @11:52AM (#18543953)

Amazon SQS is not an ideal choice because it is not designed to be a “forever” queue. Messages will expire in two weeks.

[ Parent | on HN ]

Re:Ask HN: How would you queue and process 10K+ long running... by rubenhak on Tuesday November 27, 2018 @01:48PM (#18545336)

He doesn't seem to need a forever queue. Processing might take a while, but he just needs to get it processed once (and exactly once) and move on. Yes, if he wants to do event sourcing, then yes SQS would not suffice. But I don't think he needs that. But SQS seems to be a great choice.

[ Parent | on HN ]

Re:Ask HN: How would you queue and process 10K+ long running... by bballer on Tuesday November 27, 2018 @08:23PM (#18548489)

Yeah we are already using SQS for a similar system that's currently in production. We don't need a forever queue as part of the goal is to keep the queue as empty as possible at all times. What is really important is making sure that only one job (with a distinct set of params) can be scheduled at one time and run by only worker at one time.

[ Parent | on HN ]