Ask HN: How would you queue and process 10K+ long running jobs

Re:Ask HN: How would you queue and process 10K+ long running... by superasn on Tuesday November 27, 2018 @01:47PM (#18545326)

We did something similar on our site but the max duration of each job was 5 mins. So maybe this solution may not be totally relevant to you but what were doing is we've created a aws lambda function that is triggered when a file is written on S3.

So instead of traditional SQS we just write unique files on S3 with job data and that triggers the lambda function to process the job and notify a URL upon completion.

[ Parent | on HN ]

Re:Ask HN: How would you queue and process 10K+ long running... by bballer on Tuesday November 27, 2018 @08:19PM (#18548462)

Yeah lambda functions won't suffice for the kind of jobs we run. Plus we are a full java shop and the don't want to pay for the cold start times on lambda for the jobs that do end up only taking < 1 minutes.

Thanks for your thoughts!

[ Parent | on HN ]