AWS RDS Postmortem: Is AWS Collapsing Under its Own Weight ?

Re:AWS RDS Postmortem: Is AWS Collapsing Under its Own Weight ? by ruben81adelaide on Monday May 02, 2022 @02:12AM (#31231925)

> Does Cloudwatch confirm that RDS "needed" to scale?

Actually you pointed out to a clue that I missed. I should have checked cloud watch!

The free space graphs shows that at some point something got the 200 GB of space that we originally assigned.

This is a good clue that I am going to dig in.

Thanks for your feedback!

EDIT: Cloudwatch metrics in RDS has been the key to find the source of the issue.

[ Parent | on HN ]

Re:AWS RDS Postmortem: Is AWS Collapsing Under its Own Weight ? by viraptor on Monday May 02, 2022 @05:07AM (#31232766)

So what was it? My money would be on (in order): Wal, logs, unexpected indices growth.

[ Parent | on HN ]

Re:AWS RDS Postmortem: Is AWS Collapsing Under its Own Weight ? by ruben81adelaide on Monday May 02, 2022 @05:52AM (#31233045)

The things started to go from bad to worse when work_mem parameter has not been set up correctly.

Some queries that requires a large amount of memory to process started to use disk.

Once the autoscaling kicked in, even if we would have realized about it, wouldn't have helped as you are locked out of the system.

The auto-scaling event was triggered earlier than our alerts for low disk available.

What I don't know yet, is that we have been moving services for multiple days, and the service that required the work_mem parameter was in production in that AWS for 48 hours before started to use disk rather than memory to process the SQL Queries.

[ Parent | on HN ]

Re:AWS RDS Postmortem: Is AWS Collapsing Under its Own Weight ? by synicalx on Monday May 02, 2022 @07:28PM (#31242471)

Interesting problem to come across, sounds like this is a scenario where RDS and the lack of host access/visibility was a bit of a handicap. Glad you found the issue though, hopefully things go smooth next time!

[ Parent | on HN ]