Re:AWS Frankfurt incident
by tomw1808 on Friday June 11, 2021 @06:07AM
(#27471314)
We had a few interesting "bugs" appear: Mostly logging went down, but EC2 machines kept running. E.g. We are running a lot in ECS with EC2 machines, like a RabbitMQ cluster with 20 instances all across all AZs. None of the machines died, none of the containers had degraded performance as far as I can tell, but the performance logs just disappear from 21:50 to 22:30 GMT+2. It's just blank int he Metrics and Cloudwatch dashboard. Same with other cloudwatch logs.
We also do have some EC2s running with nodejs applications and there the aws-sdk just errored out with "UnknownError: 503" and simply stopped logging until we restarted the machines. The machines itself were not stopped at all.
Other than that, I can't see any effects across our accounts. Also not RDS or anything else. Fascinating. Glad its under control and seemingly nobody died or so.
We also do have some EC2s running with nodejs applications and there the aws-sdk just errored out with "UnknownError: 503" and simply stopped logging until we restarted the machines. The machines itself were not stopped at all.
Other than that, I can't see any effects across our accounts. Also not RDS or anything else. Fascinating. Glad its under control and seemingly nobody died or so.