Post Mortem of Google Outage on 14 December 2020

Re:Post Mortem of Google Outage on 14 December 2020 by jugg1es on Friday December 18, 2020 @11:21PM (#25474994)

Ooh boy. Undocumented limitations caused major problems with us with AWS SQS. It is now in the documentation (maybe because I complained to our AWS rep), but SQS has pool of 20k messages that it will pull from when serving requests. If you are using a FIFO queue and you have 20k messages with the same message group ID in that buffer, then you are unable to process any other message regardless of how large the queue is. It caused multiple severe degradations until we rearchitected how messages are grouped. If only it had been in the documentation from day 1.

Gotta love it!

[ Parent | on HN ]

Re:Post Mortem of Google Outage on 14 December 2020 by shaicoleman on Saturday December 19, 2020 @09:16AM (#25477508)

Pretty scary that it just breaks without any error messages, and without an easy way to clean up the damage.

Queue overflows are not uncommon.

"For FIFO queues, there can be a maximum of 20,000 inflight messages (received from a queue by a consumer, but not yet deleted from the queue). If you reach this quota, Amazon SQS returns no error messages. If your queue has a large backlog of 20,000 or more messages with the same message group ID, FIFO queues might be unable to return the messages that have a different message group ID but were sent to the queue at a later time until you successfully consume the messages from the backlog."

Link to the documentation: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQS...

[ Parent | on HN ]

Re:Post Mortem of Google Outage on 14 December 2020 by jugg1es on Saturday December 19, 2020 @10:18PM (#25483536)

I only figured out what was happening by trial and error while trying to fix the production issue where I was moving 1000 messages at a time into a 2nd queue. Once I had dequeued 20k, everything starting pumping like normal. That's when I fired off an email to our AWS rep asking for confirmation from the SQS dev team. Within a couple weeks, the docs you just quoted were added.

[ Parent | on HN ]

Re:Post Mortem of Google Outage on 14 December 2020 by HeavyStorm on Saturday December 19, 2020 @08:30AM (#25477279)

Wow, that's a scary story.

[ Parent | on HN ]

Re:Post Mortem of Google Outage on 14 December 2020 by jugg1es on Saturday December 19, 2020 @10:19PM (#25483549)

Yea, we had a queue backlog of 3 million (that were bound by an SLA) by the time I figured out what was going on.

[ Parent | on HN ]