News for hackers. Stuff that matters.
Re:On SQS by mateuszf on Monday May 27, 2019 @02:52PM (#20024398)
We have a library that puts the payload in s3 bucket under random key, the bucket has expiration policy of few days. Then we generate http link to the object and send an sqs message with this url in metadata. The reader library gets data from s3, it doesn't even have to remove it. It will disappear automatically later.

We do it "by ourselves", not using the provided lib, because that way it works both for SQS and SNS. The provided lib only supports SQS.

Also our messages aren't typically very big, so we do this only if the payload size demands it.

Re:On SQS by Jemaclus on Monday May 27, 2019 @06:54PM (#20025839)
Wouldn't sending and later retrieving millions of S3 objects be expensive?
Re:On SQS by mateuszf on Tuesday May 28, 2019 @01:21AM (#20027420)
Messaging in our system is not high volume so not in this case. Also as I said - these big messages are pretty rare.
Re:On SQS by Jemaclus on Tuesday May 28, 2019 @01:54PM (#20032603)
Gotcha. That makes sense.