News for hackers. Stuff that matters.
Re:Summary of the Amazon DynamoDB Service Disruption by felixgallo on Wednesday September 23, 2015 @11:10AM (#10265478)
"It was impacting a relatively small number of customers, but we should have posted the green-i to the dashboard sooner than we did on Monday."

Amazon. No. Please listen. You should never post a green-i. Green-i means nothing to anyone. It's a minimization of a problem. You should change the indicator to show that there is a problem. If there is a problem, that is what the indicator is for. Customers don't care about whatever weird internal political pressure causes you to want to show as little information as possible on that dashboard. Customers want to know that there is a problem, and we will totally be able to figure out if it is small or large on our own.

Re:Summary of the Amazon DynamoDB Service Disruption by deanCommie on Wednesday September 23, 2015 @11:16AM (#10265512)
What's with this all-or-nothing attitude? What is so wrong about different severity levels?

Green-i: Some small percentage of customers are affected, but most of you have nothing to worry about. So if you are a customer having problems and see this notice, you are re-assured that you are not crazy. And if you are NOT seeing problems, you probably won't. Amazon has not always been the best at posting this information quickly enough, hence a commitment to be better at messaging for those customers

Yellow: A significant amount of customers is affected. Potentially serious problems, and if you haven't seen them yet, don't be surprised if you will. Start monitoring your service health and preparing for a failover to another region.

Red: FUBAR. Happens rarely, but when it does you'll probably know even before your application alarms - you'll notice when half the internet shuts down.

Re:Summary of the Amazon DynamoDB Service Disruption by felixgallo on Wednesday September 23, 2015 @11:25AM (#10265576)
different severity levels are fine. That's what the indicator is for. Decorating the indicator, normally completely incorrectly, with a tiny extra indicator, is incomprehensible. That's why you've never seen a tiny red light in the upper corner of your green traffic light. Your 'check engine' light is not green with a tiny extra indicator in the corner. The metaphors go on and on.

Customers are totally not interested in the fact that some other random customer may not be experiencing problems and that amazon has a wide variety of services, some of which may not be affected in certain geographies. Customers go to that panel with one express goal: to discover if the problems they are seeing with their servers could be related to them. It is a triage check. The question is not, is Amazon super great and are the availability engineers the bestest on average over time? The question is, what the fuck is going on?

Amazon has suffered catastrophic unavailability, and the green-i has appeared belatedly, an hour or more into the problem. Because as revealed in this post, it is manually put up by engineers. Manually!

Manually!

Re:Summary of the Amazon DynamoDB Service Disruption by deanCommie on Wednesday September 23, 2015 @12:04PM (#10265863)
The question of timing of the green-I is valid, but different from your original complaint.

In your hypothetical scenario, the customer that is going to the panel to "find out what the fuck is going on" is not going to miss the indicator i, and will read the description of the message.

Re:Summary of the Amazon DynamoDB Service Disruption by toomuchtodo on Wednesday September 23, 2015 @12:45PM (#10266154)
I don't care what that "i" indicator is; if you're showing green while you're impacted, your status board is useless and I'm going to Twitter and HN to see if you're broken (which is what I already have to do for AWS).
Re:Summary of the Amazon DynamoDB Service Disruption by mgkimsal on Wednesday September 23, 2015 @08:19PM (#10269145)
When we've seen problems, some of my clients go there and see green and say "Amazon shows they're fine - it must be your code or something you did or didn't do - fix it". Well... it's green but it's not fine. And there's little I can do to demonstrate to someone that it's not code but infrastructure.
Re:Summary of the Amazon DynamoDB Service Disruption by RyanGWU82 on Wednesday September 23, 2015 @12:36PM (#10266078)
It's really easy to miss the indicator "i". I've been doing this for 4 years and it still doesn't catch my eye when scanning the page.
Re:Summary of the Amazon DynamoDB Service Disruption by deskamess on Wednesday September 23, 2015 @12:52PM (#10266209)
I agree 100%. That column of uniform green makes it easy to miss. It is a poor UI choice but can be easily fixed.
Re:Summary of the Amazon DynamoDB Service Disruption by rstupek on Wednesday September 23, 2015 @03:07PM (#10267331)
However it is hard to miss the difference in the column with details and the more link even upon a cursory scan.
Re:Summary of the Amazon DynamoDB Service Disruption by mentat on Wednesday September 23, 2015 @02:00PM (#10266739)
> Some small percentage of customers are affected, but most of you have nothing to worry about.

That's absolutely not what it means practically for any of the recent outages. It has come to mean "the service is down but not hard enough for us to admit it".

Re:Summary of the Amazon DynamoDB Service Disruption by d23 on Wednesday September 23, 2015 @02:08PM (#10266809)
> By 2:37am PDT, the error rate in customer requests to DynamoDB had risen far beyond any level experienced in the last 3 years, finally stabilizing at approximately 55%.

Yeah, a small green-i should cover it.

Re:Summary of the Amazon DynamoDB Service Disruption by nickpsecurity on Wednesday September 23, 2015 @01:48PM (#10266628)
Green traditionally means we're good, ready to go, etc. Non-intuitive to use any green indicator for a situation deserving a write-up like this. It's yellow at best.
Re:Summary of the Amazon DynamoDB Service Disruption by cthalupa on Wednesday September 23, 2015 @02:47PM (#10267123)
During the actual outage this write up was about, didn't it go red?
Re:Summary of the Amazon DynamoDB Service Disruption by nickpsecurity on Wednesday September 23, 2015 @04:16PM (#10267845)
I wasn't watching the indicator. I started with the parent comment's assertion. If it was wrong, my comment doesn't apply to this outage and becomes a rule of thumb.
Re:Summary of the Amazon DynamoDB Service Disruption by takeda on Wednesday September 23, 2015 @01:52PM (#10266664)
Yep, especially yellow is labeled clearly as "Performance issues". It is basically lying to make things appear better than they really are.
Re:Summary of the Amazon DynamoDB Service Disruption by jamiesonbecker on Wednesday September 23, 2015 @03:56PM (#10267706)
I'm starting to wonder how much more I should trust AWS. Is everything swept under the carpet?

e.g., not even attributed or thanks for root escalation vulnerabilities that I surfaced for them: https://news.ycombinator.com/item?id=10261507