News for hackers. Stuff that matters.
Re:On SQS by Jemaclus on Monday May 27, 2019 @01:05PM (#20023582)
We love SQS, but one of the problems we're running into lately is the 256kb per message limitation. We do tens of millions of messages per day, with a small percentage of those reaching the 256kb limit. We're approaching the point where most of our messages will hit that limit.

What are our options for keeping SQS but somehow sending large payloads? Only thing I can think of is throwing them into another datastore and using the SQS just as a pointer to, say, a key in a Redis instance.

(Kafka is probably off the table for this, but I could be convinced. I'd like to hear other solutions first, though.

Re:On SQS by m4r on Monday May 27, 2019 @01:19PM (#20023711)
Hard to suggest smth without details, but imo 256kb per message is a lot and enough. One thing I would consider is storing such "messages" payload on s3, and sending notification to sqs allowing you to fetch given file and process it (this however might be expensive if you happen to have such high traffic)
Re:On SQS by mateuszf on Monday May 27, 2019 @02:52PM (#20024398)
We have a library that puts the payload in s3 bucket under random key, the bucket has expiration policy of few days. Then we generate http link to the object and send an sqs message with this url in metadata. The reader library gets data from s3, it doesn't even have to remove it. It will disappear automatically later.

We do it "by ourselves", not using the provided lib, because that way it works both for SQS and SNS. The provided lib only supports SQS.

Also our messages aren't typically very big, so we do this only if the payload size demands it.

Re:On SQS by Jemaclus on Monday May 27, 2019 @06:54PM (#20025839)
Wouldn't sending and later retrieving millions of S3 objects be expensive?
Re:On SQS by mateuszf on Tuesday May 28, 2019 @01:21AM (#20027420)
Messaging in our system is not high volume so not in this case. Also as I said - these big messages are pretty rare.
Re:On SQS by Jemaclus on Tuesday May 28, 2019 @01:54PM (#20032603)
Gotcha. That makes sense.
Re:On SQS by mark242 on Monday May 27, 2019 @01:22PM (#20023733)
The Java SDK for SQS automatically drops >256k messages into an S3 bucket and stores a pointer to the message in SQS itself. You set the bucket and the client transparently handles retrieving a message from SQS/S3 when necessary.

Based on the structure of the message (UUIDv4) you could probably roll your own implementation in any language.

Re:On SQS by Jemaclus on Monday May 27, 2019 @06:54PM (#20025842)
That's a good idea, but wouldn't sending and later retrieving millions of S3 objects be expensive?
Re:On SQS by manigandham on Monday May 27, 2019 @08:20PM (#20026212)
Yes, and slow, if you're actually doing it for millions. The comment was saying the library only does it for messages that are over the size limit though.
Re:On SQS by Jemaclus on Tuesday May 28, 2019 @01:15AM (#20027396)
Gotcha.
Re:On SQS by manish_gill on Monday May 27, 2019 @03:58PM (#20024860)
I've successfully been using Kafka in production with a tiny percentage of packets ~1Mb. We utilize gzip compression with chunking on some of those messages (system is legacy, no fancy compressed message format in use). IIRC, only had to modify settings at the broker level and it works perfectly fine.
Re:On SQS by Jemaclus on Monday May 27, 2019 @06:56PM (#20025859)
We've discussed switching to Kafka. There are some pros/cons to doing that. With respect to my problem above, our messages _could _conceivably approach 1MB (or even surpass it), so we're really just delaying the inevitable. That said, we're a long, long way from hitting that limit, so it's definitely something we're looking at.

We just recently started gzipping our payloads, which buys us even more time.

Re:On SQS by TeeWEE on Tuesday May 28, 2019 @09:56AM (#20030016)
Why are your messages large? Messages should just be signalling events, like follows:

OrderCreated(order_id=23)

UserLoggedIn(user_id=342)

etc.

I guess you are sending the whole user object over the event bus? Isnt that an anti-pattern?

Re:On SQS by somedev55 on Monday May 27, 2019 @01:10PM (#20023633)
We point to an s3 object for any large payloads
Re:On SQS by Jemaclus on Monday May 27, 2019 @06:54PM (#20025838)
Wouldn't sending and later retrieving millions of S3 objects be expensive?
Re:On SQS by 8note on Monday May 27, 2019 @01:13PM (#20023657)
and there's a library around that can do it for you, at least in java