Re:Show HN: Hivemind – Distributed jobs using AWS Lambda fun...
by teej on Thursday September 28, 2017 @07:09PM
(#15361539)
What we do is a separate Python process, not in Lambda, that polls status. Status is persisted in Redis. It supports lambdas calling other lambdas so dynamic workflows just work and can live for longer than 300s. This has been working fine so far but would probably have issues at 10k+.
Re:Show HN: Hivemind – Distributed jobs using AWS Lambda fun...
by tyingq on Thursday September 28, 2017 @10:12PM
(#15362272)
There's an SQS/Lamba integration. Push 10k messages in a queue as jobs. The lambdas pull them off and delete them once done.
https://cloudonaut.io/integrate-sqs-and-lambda-serverless-ar...
Re:Show HN: Hivemind – Distributed jobs using AWS Lambda fun...
by teej on Friday September 29, 2017 @02:14AM
(#15363138)
From what I can tell, this doesn't really help you know when your queued up lambdas have all completed, which is what GP wants.
Re:Show HN: Hivemind – Distributed jobs using AWS Lambda fun...
by tyingq on Friday September 29, 2017 @07:46AM
(#15364436)
You're right. I assumed the queue depth api call would be useable as a counter. It appears though, that it's only an approximate count unless you use FIFO queues.
I guess the individual Lambdas could pop an "I'm done" message onto a FIFO queue. With a single Lambda consumer of that queue that updates an S3 bucket with a count. Done when count == 10,000. That's atomic, and no polling, but feels a bit smelly.
Re:Show HN: Hivemind – Distributed jobs using AWS Lambda fun...
by joseph on Thursday September 28, 2017 @07:03PM
(#15361513)
Have you tried Step Functions? I've used them to orchestrate Lambdas with good success when I needed something that will run longer than Lambdas can individually.
Re:Show HN: Hivemind – Distributed jobs using AWS Lambda fun...
by kondro on Thursday September 28, 2017 @08:01PM
(#15361770)
Yeah, they’re good, but more expensive than you’d expect given the relative pricing compared to other AWS services. $0.025/1000 transitions seems and order of magnitude or two high given it’s just ultimately a central job store with some very, very simple logic for changing state.
Re:Show HN: Hivemind – Distributed jobs using AWS Lambda fun...
by tfjaeckel on Friday September 29, 2017 @04:44PM
(#15369022)
Step Functions are convenient but for the reason you stated we stopped using them. Pricing seems off for what they do, especially when it's in volume.
Re:Show HN: Hivemind – Distributed jobs using AWS Lambda fun...
by btashton on Thursday September 28, 2017 @08:01PM
(#15361774)
I wrote a simple lambda function that takes in a payload and a step function arn. It then fires up a bunch of step function jobs and maps the execution identifier to the input and stores that in s3 for later access. This ends up being really nice for creating distributed cron like jobs by just connecting the lambda to cloudwatch events.
I would really like AWS to have an event that fires when all the lambdas in a set have completed, but that doesn't seem to exist. Any suggestions?