Re:New for AWS Lambda: Use Any Programming Language and Shar...
by scarface74 on Thursday November 29, 2018 @10:08PM
(#18566311)
I usually don’t let the lambda SQS back end control message deletion for just that reason. I delete individual messages as I process them and throw an exception if any fail.
Re:New for AWS Lambda: Use Any Programming Language and Shar...
by otterley on Friday November 30, 2018 @02:07AM
(#18567325)
What problems have you encountered using the same SNS topic as the dead letter queue for the Lambda function?
Re:New for AWS Lambda: Use Any Programming Language and Shar...
by philwelch on Friday November 30, 2018 @06:53PM
(#18573904)
I haven’t even tried that because that sounds like an infinite retry scenario waiting to happen.
Re:New for AWS Lambda: Use Any Programming Language and Shar...
by otterley on Sunday December 02, 2018 @04:38PM
(#18584714)
Indeed it is - I was assuming, perhaps mistakenly, that that was the desired behavior. After all, why dispatch requests that are not to be processed?
Re:New for AWS Lambda: Use Any Programming Language and Shar...
by philwelch on Monday December 03, 2018 @12:35PM
(#18590772)
If a specific message fails to process multiple times, most often it’s because that particular message is malformed or invalid, or exposes an unhandled edge case in the processing logic, and will never successfully process, at which point continued attempts are both fruitless and wasteful.
It’s better to retry a finite number of times and stash any messages that fail to process offline for further investigation. If there is an issue that can be resolved, it’s possible to feed deadletter messages back in for processing once the issue is addressed.
Feeding your deadletter events back into your normal processing flow not only hides these issues but forces you to pay real money to fail to process them in an infinite loop that is hard if not impossible to diagnose and detect.
It's also a useful pattern to, instead of configuring the SQS event itself as your Lambda trigger, have a scheduled trigger and then perform health checks on your service dependencies before pulling messages from the queue. This technique saves you wasted retries and even gives you more fine-grained control over deleting messages (the SQS mechanism for saying, "this is done and you don't need to retry it after the timeout). The SQS trigger, in contrast, deletes the entire batch of messages if the Lambda exits cleanly and does not delete any of them if the Lambda does not exit cleanly, which is a bit messy.
Also, you can have an SQS queue subscribe to multiple SNS topics, which means you can set up separate SNS topics for backfill or other use cases. This is especially useful in cross-account use cases, where often your online events are coming from an SNS from a different account, but you want to be able to inject your own testing and backfill events.