News for hackers. Stuff that matters.

A comment on Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region. Parent comment

Re:Summary of the AWS Service Event in the Northern Virginia... by luhn on Friday December 10, 2021 @06:54PM (#29516943)
Off the top of my head, this is the third time they've had a major outage where they've been unable to properly update the status page. First we had the S3 outage, where the yellow and red icons were hosted in S3 and unable to be accessed. Second we had the Kinesis outage, which snowballed into a Cognito outage, so they were unable to login into the status page CMS. Now this.

They "own up to it" in their postmortems, but after multiple failures they're still unwilling to implement the obvious solution and what is widely regarded as best practice: host the status page on a different platform.

[ Parent | on HN ]
Re:Summary of the AWS Service Event in the Northern Virginia... by 46Bit on Friday December 10, 2021 @10:06PM (#29518030)
Firmly agreed. I've heard AWS discuss making the status page better – but they get really quiet about actually doing it. In my experience the best/only way to check for problems is to search Twitter for your AWS region name.
[ Parent | on HN ]
Re:Summary of the AWS Service Event in the Northern Virginia... by ngc248 on Saturday December 11, 2021 @02:24AM (#29519210)
Maybe AWS should host their status checks in Azure and vice versa ... Mutually Assured Monitoring :) Otherwise it becomes a problem of who will monitor the monitor
[ Parent | on HN ]
Re:Summary of the AWS Service Event in the Northern Virginia... by isbvhodnvemrwvn on Saturday December 11, 2021 @09:25AM (#29521054)
My company is quite well known for blameless post-mortems, but if someone failed to implement improvements after three subsequent outages, they would be moved to a position more appropriate for their skills.
[ Parent | on HN ]
Re:Summary of the AWS Service Event in the Northern Virginia... by koheripbal on Friday December 10, 2021 @08:57PM (#29517690)
This challenge is not specific to Amazon.

Being able to automatically detect system health is a non-trivial effort.

[ Parent | on HN ]
Re:Summary of the AWS Service Event in the Northern Virginia... by bob778 on Friday December 10, 2021 @09:00PM (#29517701)
That’s not what’s being asked though - in all 3 events, they couldn’t manually update it. It’s clearly not a priority to fix it for even manual alerts.
[ Parent | on HN ]
Re:Summary of the AWS Service Event in the Northern Virginia... by jjoonathan on Friday December 10, 2021 @09:11PM (#29517758)
They had all day to do it manually.
[ Parent | on HN ]
Re:Summary of the AWS Service Event in the Northern Virginia... by moralestapia on Friday December 10, 2021 @09:34PM (#29517896)
>Be capable of spinning up virtualized instances (including custom drive configurations, network stacks, complex routing schemes, even GPUs) with a simple API call

But,

>Be incapable of querying the status of such things

Yeah, I don't believe it.

[ Parent | on HN ]
Re:Summary of the AWS Service Event in the Northern Virginia... by saagarjha on Friday December 10, 2021 @09:18PM (#29517797)
As others mention, you can do it manually. But it’s also not that hard to do automatically: literally just spin up a “client” of your service and make sure it works.
[ Parent | on HN ]
Re:Summary of the AWS Service Event in the Northern Virginia... by blackearl on Friday December 10, 2021 @09:00PM (#29517702)
Why automatic? Surely someone could have the responsibility to do it manually.
[ Parent | on HN ]
Re:Summary of the AWS Service Event in the Northern Virginia... by geenew on Friday December 10, 2021 @10:20PM (#29518097)
Or override the autogenerated values
[ Parent | on HN ]
Re:Summary of the AWS Service Event in the Northern Virginia... by thebean11 on Friday December 10, 2021 @06:56PM (#29516957)
Eh, the colored icons not loading is not really the same thing as incorrectly reporting that nothing’s wrong. Putting the status page on separate infra would be good practice, though.
[ Parent | on HN ]
Re:Summary of the AWS Service Event in the Northern Virginia... by dijit on Friday December 10, 2021 @07:00PM (#29516992)
The icons showed green.
[ Parent | on HN ]