Ask HN: How would you queue and process 10K+ long running jobs

Re:Ask HN: How would you queue and process 10K+ long running... by dmarlow on Tuesday November 27, 2018 @11:11AM (#18543532)

Can you elaborate on what the "job" is or does? Can things be batched?

[ Parent | on HN ]

Re:Ask HN: How would you queue and process 10K+ long running... by bballer on Tuesday November 27, 2018 @08:33PM (#18548547)

I wont fully elaborate on what a job is but I'll give you a couple examples:

Updating anywhere from 500-5M items over APIs that are rate limited.

Dumping datasets that have to be normalized and massaged into files/ and dropped off at third party servers anywhere from every 15 minutes to once a month. These files could contain anywhere from 500 lines to 5M lines.

Injesting datasets just as large as described above but massaged and saved into our caches and DB.

[ Parent | on HN ]