Ask HN: How would you queue and process 10K+ long running jobs

Re:Ask HN: How would you queue and process 10K+ long running... by bballer on Tuesday November 27, 2018 @08:21PM (#18548469)

For workers we run only Java and run everything on AWS. We have an existing production system that uses a combination of SQS, DynamoDB, and Postgres, and EC2s to achieve something very similar to this. Just want to check all the boxes before we dive into building out something for a new system coming into production that shares many of the same requirements.

[ Parent | on HN ]

Re:Ask HN: How would you queue and process 10K+ long running... by rubenhak on Wednesday November 28, 2018 @09:38PM (#18557606)

SQS-FIFO should let you process task once & only once. Just make sure you configure timing parameters correctly.

DynamoDB has triggers that get fired upon changes. That would strongly help with eventual consistency implementation (which i strongly recommend). But with this you should write Lambdas. Check how well is Lambda Java supported.

Are you sure you want to use EC2 directly? Why not to use ECS? This would let you focus more on the business and less on infrastructure

[ Parent | on HN ]