They "own up to it" in their postmortems, but after multiple failures they're still unwilling to implement the obvious solution and what is widely regarded as best practice: host the status page on a different platform.
Being able to automatically detect system health is a non-trivial effort.
But,
>Be incapable of querying the status of such things
Yeah, I don't believe it.
This seems like bad faith to me based on my experience when I worked for AWS. As they repeated many times at Re:Invent last week, they've been doing this for 15+ years. I distinctly remember seeing banners like "Don't update the dashboard without approval from <importantSVP>" on various service team runbooks. They tried not to say it out loud, but there was very much a top-down mandate for service teams to make the dashboard "look green" by:
1. Actually improving availability (this one is fair).
2. Using the "Green-I" icon rather than the blue, orange, or red icons whenever possible.
3. They built out the "Personal Health Dashboard" so they can post about many issues in there, without having to acknowledge it publicly.
How'd that work out for them?
https://duckduckgo.com/?q=AWS+outage+news+coverage&t=h_&ia=w...
It's smart politics -- I don't blame them but I don't trust the dashboard either. There's established patterns now of the AWS dashboard being useless.
If I want to check if Amazon is down I'm checking Twitter and HN. Not bad faith -- no faith.
Um, so you think straight-up lying is good politics?
Any 7-year old knows that telling a lie when you broke something makes you look better superficially, especially if you get away with it.
That does not mean that we should think it is a good idea to tell lies when you break things.
It sure as hell isn't smart politics in my book. It is straight-up disqualifying to do business with them. If they are not honest about the status or amount of service they are providing, how is that different than lying about your prices?
Would you go to a petrol station that posted $x.00/gallon, but only delivered 3 quarts for each gallon shown on the pump?
We're being shortchanged and lied to. Fascinating that you think it is good politics on their part.
AWS spends a lot of time thinking about this problem in service to their customers.
How do you reduce the status of millions of machines, the software they run, and the interconnected-ness of those systems to a single graphical indicator?
It would be dumb and useless to turn something red every single time anything had a problem. Literally there are hundreds of things broken every minute of every day. On-call engineers are working around the clock on these problems. Most of the problems either don’t affect anyone due to redundancy or affect only a tiny number of customers- a failed memory module or top-of-rack switch or a random bit flip in one host for one service.
Would it help anyone to tell everyone about all these problems? People would quickly learn to ignore it as it had no bearing on their experience.
What you’re really arguing is that you don’t like the thresholds they’ve chosen. That’s fine, everyone has an opinion. The purpose of health dashboards like these are mostly so that customers can quickly get an answer to “is it them or me” when there’s a problem.
As others on this thread have pointed out, AWS has done a pretty good job of making the SHD align with the subjective experience of most customers. They also have personal health dashboards unique to each customer, but I assume thresholding is still involved.
A good low-hanging fruit would be, when the outage is significant enough to have reached the media, you turn the dot red.
Dishonesty is what we're talking about here. Not the gradient when you change colors. This is hardly the first major outage where the AWS status board was a bald-faced lie. This deserves calling out and shaming the responsible parties, nothing less, certainly not defense of blatantly deceptive practices that most companies not named Amazon don't dip into.
Broken dashboard != lying.
The specific charge of “lying” is what I dispute.
There's a limitless variety of options, and multiple books written about it. I can recommend the series "The Visual Display of Quantitative Information" by Edward Tufte, for starters.
>> Literally there are hundreds of things broken every minute of every day. On-call engineers are working around the clock...
Of course there are, so a single R/Y/G indicator is obviously a bad choice.
Again, they could at any time easily choose a better way to display this information, graphs, heatmaps, whatever.
More importantly, the one thing that should NOT be chosen is A) to have a human in the loop of displaying status, as this inserts both delay and errors.
Worse yet, to make it so that it is a VP-level decision, as if it were a $1million+ purchase, and then to set the policy to keep it green when half a continent is down... ummm that is WAAAYYY past any question of "threshold" - it is a premeditated, designed-in, systemic lie.
>>You don’t know what you’re talking about. Look in the mirror, dude. While I haven't worked inside AWS, I have worked in complex network software systems and well understand the issues of thousands of HW/SW components in multiple states. More importantly, perhaps it's my philosophy degree, but I can sort out WHEN (e.g., here) the problem is at another level altogether. It is not the complexity of the system that is the problem, it is the MANAGEMENT decision to systematically lie about that complexity. Worse yet, it looks like those lies on an everyday basis are what goes into their claims of "99.99+% uptime!!" evidently false. The problem is at the forest level, and you don't even want to look at the trees because you're stuck in the underbrush telling everyone else they are clueless.
it's a bullshit board used fudge numbers when negoaiting SLAs
like I don't care that much, hell my company does the same thing; but let's not get defensive over it
Twice is a coincidence.
Three times is a pattern.
But this… This is every time.
The problem is that the dashboard requires VP approval to be updated. Which is broken. The dashboard should be automatic. The dashboard should update before even a single member of the AWS team knows there's something wrong.
Let's not act like this is the first time this has happened. It's bad faith that they do not change when their promise is they hire the best to handle infrastructure so you don't have to. It's clearly not the case. Between this and billing I we can easily lay blame and acknowledge lies.
They have a strong incentive to lie, and they're doing it. This makes people dependent upon the truth for refunds understandably angry.
Because Amazon kills industries. Takes job. They do this because they promise they hire the best people that can do this better than you and for cheaper. And it's rarely true. And then they lie about it when things hit the fan. If you're going to be the best you need to act like the best, and execute like the best. Not build a walled garden that people cant see into, and hard to leave.
The comments are frustrated users.
Not hateful.
It seems that degraded seems down in most cases. Since authorization of managers is required.
This to me is really bad. Even as a small company, we keep our support infrastructure separate. For a company of Amazon's size, this is a shitty excuse. If I cannot even reach you as a customer for almost 7 hours, that is just nuts. AWS must do better here.
Also, is it true that the outage/status pages are manually updated ? If yes, there is no excuse why it was green for that long. If you are manually updating it, please update asap.
there's such as thing as too much dogfooding.
It turned out our call center supplier had something running on AWS, and it took out our entire phone system. After this situation settles, I'm tempted to ask my supplier to see what they're doing to get around this in the future, but I doubt even they knew that AWS was used further downstream.
AWS operates a lot like Amazon.com, the marketplace now--you can try to escape it, but it's near impossible. If you want to ban usage of Amazon's services, you're going to find some service (AWS) or even a Shopify site (FBA warehouse) who uses it.
My favourite is when some company migrates their physical servers to virtual machines, including the AD domain controllers. Then the next step is to use AD LDAP authentication for the VM management software.
When there's a temporary outage and the VMs don't start up as expected, the admins can't log on and troubleshoot the platform because the logon system was running on it... but isn't now.
The loop is closed.
You see this all the time, especially with system-management software. They become dependent on the systems they're managing, and vice-versa.
If you care about availability at all, make sure to have physical servers providing basic services like DNS, NTP, LDAP, RADIUS, etc...
One of our auxiliary services that's basically a pass through to AWS was offline nearly the entire day. Yet, this communication doesn't even mention that fact. In fact, it almost tries to suggest the opposite.
Likewise, AWS is reporting S3 didn't have issues. Yet, for a period of time, S3 was erroring out frequently because it was responding so slowly.
SLAs that refund only the cost of the individual service that was down is worthless.
SLAs that require separate proof and refund requests for each and every service that was affected are nearly worthless.
There needs to be an independent body set up by a large cloud customers to monitor availability and enforce refunds.
Unless you're talking about some board other than the Service Health Dashboard, this isn't true. They dropped EC2 down to degraded pretty early on. I bemusedly noted in our corporate Slack that every time I refreshed the SHD, another service was listed as degraded. Then they added the giant banner at the top. Their slight delay in updating the SHD at the beginning of the outage is mentioned in the article. It was absolutely not all green for the duration of the outage.
1 hour (52 minutes according to the article) sounds meh. I wonder what their error rate and latency graphs look like from that day.
They've discovered it right away, the Service Health Dashboard was not updated. source: link.
But yes you’re right, there’s no reason to question their monitoring or alerting specifically.
Sagemaker, for example, was down all day. I was dead in the water on a modeling project that required GPUs. It relied on EC2, but nobody there even thought to update the status? WTF. This is clearly executives incentivized to let a bug persist. This is because the bug is actually a feature for misleading customers and maximizing profits.
They are barely doing better than 2 9s.
My boss had not given me any special directions here so I assumed I was supposed to do this honestly. I set our status as "Red" and then listed out what were, I felt, quite compelling reasons to think we were Red. The gist of it was that our velocity was negative. More work items were getting created and assigned to us than we closed, and we still had high priority items open from previous dates. There was zero chance, in my estimation, that we would meet our deadlines, so I called us Red.
This did not go over well. Everyone at the Launch Readiness meeting got mad at me for declaring Red. Our VP scolded me in front of the entire meeting and lectured me about how I could not unilaterally declare our team red. Her logic was, if our team was Red, that meant the entire project was Red, and I was in no position to make that call. Other managers at the meeting got mad at me too because they felt my call made them look bad. For the rest of my manager's absence I had to first check in with a different manager and show him my Launch Readiness status and get him to approve my update before I was allowed to show it to the rest of the group.
For the rest of the time that I went to Launch Readiness I was forbidden from declaring Red regardless of what our metrics said. Our team was Yellow or Green, period.
Naturally, we wound up being over a year late on the deadlines, because, despite what they compelled us to say in those meetings, we weren't actually getting the needed work done. Constant "schedule slips" and adjustments. Endless wasted time in meetings trying to rework schedules that would instantly get blown up again. Hugely frustrating. Still slightly bitter about it.
Anyway, I guess all this is to say that it doesn't surprise me that Amazon is bad about declaring Red, Yellow, or Green in other places too. Probably there is a guy in charge of updating those dashboards who is forbidden from changing them unless they get approval from some high level person and that person will categorically refuse regardless of the evidence because they want the indicators to be Green.
In essence - what you were saying is your boss lied the whole time, because how does one go from a presumed positive velocity to negative velocity in a week?
Additionally assuming you're a dev lead, it's a little surprising that this is your first meeting of this sorts. As dev lead, I didn't always attend them but my input is always sought on the status.
Sounds like you had a bad manager, and Amazon is filled with them.
1) If you keep status green for 5 years, while not delivering anything, the reality is the folks at the very top (who can come and go) just look at these colors and don't really get into the project UNLESS you say you are red :)
2) Within 1-2 years there is always going to be some excuse for WHY you are late (people changes, scope tweaks, new things to worry about, covid etc)
3) Finally you are 3 years late, but you are launching. Well, the launch overshadows the lateness. Ie, you were green, then you launched, that's all the VP really sees sometime.
Equipment was constantly breaking down, in ways that ranged from inconvenient to potentially dangerous. Seemingly basic design decisions, like the shape of chutes, were screwed up in mind-boggling ways (they put a right-angle corner partway down each chute, which caused packages to get stuck in the chutes constantly). We were short on equipment almost every day; things like poles to help us un-jam packages were in short supply, even though we could move hundreds of thousands of packages a day. On top of all this, the facility opened with half its sorting equipment, and despite promises that we'd be able to add the rest of the equipment in the summer, during Amazon's slow season...it took them two years to even get started.
And all the while, they demanded ever-increasing package quotas. At first, 120,000 packages/day was enough to raise eyebrows--we broke records on a daily basis in our first holiday rush--but then, they started wanting 200,000, then 400,000. Eventually it came out that the building wouldn't even be breaking even until it hit something like 500,000.
As we scaled up, things got even worse. None of the improvements that workers suggested to management were used, to my knowledge, even simple things like adding an indicator light to freight elevators.
Meanwhile, it eventually became clear that there wasn't enough space to store cargo containers in the building. 737s and the like store packages mostly in these giant curved cargo containers, and we needed them to be locked in place while working around/in them...except that, surprise, the people planning the building hadn't planned any holding areas for containers that weren't in use! We ended up sticking them in the middle of the work area.
Which pissed off the upper management when they visited. Their decision? Stop doing it. Are we getting more storage space for the cans? No. Are we getting more workers on the airplane ramp so we can put these cans outside faster? No. But we're not allowed to store those cans in the middle of the work area anymore, even if there aren't any open stations with working locks. Oh, by the way, the locking mechanisms that hold the cans in place started to break down, and to my knowledge they never actually fixed any of the locks. (A guy from their safety team claims they've fixed like 80 or 90 of the stations since the building opened, but none of the broken locks I've seen were fixed in the 2 years I worked there.)
In organisations with 100s of in-flight projects, it’s understandable that red is reserved for projects that are causing extremely serious issues right now. Otherwise, so many projects would be red, that you’d need a new colour.
I think it's more like the planning people get rewarded for creating plans that look good and it doesn't bother them if the plans are unrealistic. Then, levels of middle management don't want to make themselves look bad by saying they're behind. And, ultimately, everyone figures they can play a kind of schedule-chicken where everyone says they're green or yellow until the last possible second, hoping that another group will raise a flag first and give you all more time while you can pretend you didn't need it.
Jeez, how many meetings did you go to, and how long was this person's vacation? I'm jelly of being allowed to take that much time off continuously.
"How dare you threaten our launch readiness go/no-go?!"
> While AWS customer workloads were not directly impacted from the internal networking issues described above, the networking issues caused impact to a number of AWS Services which in turn impacted customers using these service capabilities. Because the main AWS network was not affected, some customer applications which did not rely on these capabilities only experienced minimal impact from this event.
Yeah so vanilla LB and EC2 with no autoscaling were fine. Anyone using "serverless" or managed services had a real bad day
Bullshit someone at Amazon doesn’t have time to update the status.
Edit: not to mention APIGW and Cloudwatch APIs were down too.
if ($pain > $gain) {
move_your_shit_and_exit_aws();
}
sub move_your_shit_and_exit_aws
{
printf("Dude. We have too much pain. Start moving\n");
printf("Yeah. That won't happen, so who cares\n");
exit(1);
}
Of course problems and outages are going to happen, but saying they have five nines (99.999) uptime as measured by their "green board" is meaningless. During the event they were late and reluctant to report it and its significance. My point is that they are wrongly incentivized to keep the board green at all costs.