If you want sequential processing of the data in the SQS queue, something which works really well today is to create a state machine in AWS Step Functions which triggers a AWS Lambda function which then pulls data from SQS and processes it. Using a condition in the state machine, this can be done in a loop, so when the AWS Lambda function reaches its timeout, another one gets triggered as long as there is still data in the SQS queue.
If data doesn't have to be processed sequentially an option is to configure the AWS Lambda function to get invoked for new data in the SQS queue [1], so you don't have to care about manually fetching data from SQS at all.
[1]: https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html
Re:AWS Lambda pricing now per ms
by staticassertion on Tuesday December 01, 2020 @01:38PM
(#25268260)
Yep, but I prefer to care about manually fetching data from SQS. It's a weird system, but due to our data model there are many benefits to processing as many messages in a given lambda as possible.
This might be useful to simplify that model for you :)
https://aws.amazon.com/about-aws/whats-new/2020/11/aws-lambd...
Re:AWS Lambda pricing now per ms
by staticassertion on Tuesday December 01, 2020 @09:40PM
(#25272691)
Pretty cool, but not quite the model I want.
Today, for a nonstop workload you might save money with Fargate or ECS.
Re:AWS Lambda pricing now per ms
by staticassertion on Tuesday December 01, 2020 @01:37PM
(#25268245)
Yeah, I'd be curious to see how much money we'd save with Fargate.
I'm going to assume since you're processing off a queue, the lambdas are not serving a waiting user. If you do the work to orchestrate onto Fargate, you might as well go the rest of the way and move to ECS. Then you can use spots, where real savings kick in.
Someone mentioned step functions above. All of our steps run on spots. We also have some tasks like you have that read off queues and do processing, which also all run on spots.
Re:AWS Lambda pricing now per ms
by jeffcarter on Tuesday December 01, 2020 @02:02PM
(#25268555)
For raw compute, Fargate is ~1/2 the cost of Lambda. But you'd have to orchestrate the launch yourself. It could be worth it though depending on your workload
What I care about is: * Scale to 0, and automatic scaling up without configuring it
* Automatic patching of the OS
* Fault isolation
Lambda gives me that. So each one runs for 15 minutes, processing all data in an SQS queue.
I do wonder if Fargate would be cheaper per millisecond? Dunno.