News for hackers. Stuff that matters.
Re:Ask HN: Is S3 down? by fletom on Tuesday February 28, 2017 @01:16PM (#13756111)
what's truly incredible is that S3 has been offline for h̶a̶l̶f̶ ̶a̶n̶ ̶h̶o̶u̶r̶ two hours now and Amazon still has the audacity to put five shiny green checkmarks next to S3 on their service page.

they just now put up a box at the top saying "We are investigating increased error rates for Amazon S3 requests in the US-EAST-1 Region."

increased error rates? really?

Amazon, everything is on fire. you are not fooling anyone

edit: in the future, please subscribe to @MyFootballNow for timely AWS service status updates https://pbs.twimg.com/media/C5xdm9_WMAAY7y_.jpg:large

Re:Ask HN: Is S3 down? by idlewords on Tuesday February 28, 2017 @01:27PM (#13756276)
@mikecb on Twitter explained it well. "The red icon is stored in S3 US East."
Re:Ask HN: Is S3 down? by jasoncrawford on Tuesday February 28, 2017 @02:21PM (#13756905)
Apparently this is not a joke:

“The dashboard not changing color is related to S3 issue. See the banner at the top of the dashboard for updates.”

https://twitter.com/awscloud/status/836656664635846656

Re:Ask HN: Is S3 down? by noir_lord on Tuesday February 28, 2017 @02:26PM (#13756943)
I think that might be my favourite tweet this year so far.
Re:Ask HN: Is S3 down? by mpetrovich on Tuesday February 28, 2017 @02:13PM (#13756812)
There are some real gems @Pinboard too: "Green checkmark = no lava in data center. Green checkmark with information icon = data center filling with lava https://status.aws.amazon.com"
Re:Ask HN: Is S3 down? by paulddraper on Tuesday February 28, 2017 @02:02PM (#13756675)
FYI, in seriousness you can see the fabled red status icon here:

https://status.aws.amazon.com/images/status3.gif

It does exist, apparently.

Re:Ask HN: Is S3 down? by rsynnott on Tuesday February 28, 2017 @02:14PM (#13756832)
It showed up during the big DynamoDB outage last year.
Re:Ask HN: Is S3 down? by busterarm on Tuesday February 28, 2017 @02:19PM (#13756875)
Is the manager of that group still working there post-outage?
Re:Ask HN: Is S3 down? by tlrobinson on Tuesday February 28, 2017 @02:22PM (#13756910)
I thought this was a joke, but apparently not too far off:

"The dashboard not changing color is related to S3 issue. See the banner at the top of the dashboard for updates."

https://twitter.com/awscloud/status/836656664635846656

Re:Ask HN: Is S3 down? by scarlac on Tuesday February 28, 2017 @01:42PM (#13756423)
While that may be true, that's not the reason you're seeing green. You should have been seeing a broken image or a status page not finishing loading if that was an issue.
Re:Ask HN: Is S3 down? by eknkc on Tuesday February 28, 2017 @01:47PM (#13756484)
Re:Ask HN: Is S3 down? by eherot on Tuesday February 28, 2017 @02:22PM (#13756908)
Re:Ask HN: Is S3 down? by mikecb on Tuesday February 28, 2017 @04:10PM (#13757866)
The best jokes always have a grain of truth.
Re:Ask HN: Is S3 down? by koolba on Tuesday February 28, 2017 @01:58PM (#13756631)
If this is how they're going to handle an outage of their premier AWS service, it's other cloud providers that will be seeing green.
Re:Ask HN: Is S3 down? by jacquesm on Tuesday February 28, 2017 @02:06PM (#13756726)
Funny, cloudflare is also having trouble, page only showed on the third request.

Perfect storm.

As for other cloud providers seeing green: Or maybe people will come to their senses and will see that monocultures are bad, whether in biology or hosting.

Re:Ask HN: Is S3 down? by koolba on Tuesday February 28, 2017 @02:14PM (#13756826)
> Funny cloudflare is also having trouble, page only showed on the third request.

I bet they're related. The moment I got an alert of the S3 outage I started refreshing a bunch of status pages at a feverish pitch. Multiply that by a thousands of others doing the same and boom you've got the equivalent of a DDOS.

Re:Ask HN: Is S3 down? by leesalminen on Tuesday February 28, 2017 @02:28PM (#13756965)
dang [0] just commented to say that their server is getting hammered because of this thread.

[0] https://news.ycombinator.com/item?id=13756819

Re:Ask HN: Is S3 down? by maxerickson on Tuesday February 28, 2017 @02:16PM (#13756843)
I've been getting intermittent Bad Gateways on HN for the last few days.

Ray ID: 33863460edf54231

Re:Ask HN: Is S3 down? by general_failure on Tuesday February 28, 2017 @01:49PM (#13756515)
That was (obviously) sarcasm :)
Re:Ask HN: Is S3 down? by tonyedgecombe on Tuesday February 28, 2017 @02:10PM (#13756777)
Humour is wasted on the internet.
Re:Ask HN: Is S3 down? by mikecb on Tuesday February 28, 2017 @04:11PM (#13757874)
or was it? dun dun
Re:Ask HN: Is S3 down? by ckozlowski on Tuesday February 28, 2017 @02:32PM (#13756997)
(Disclaimer: I work for AWS.)

The dashboard is not changing color due to the S3 issue. We're updating the banner in place of that.

Edit: Update at 11:35 AM PST: We have now repaired the ability to update the service health dashboard. The service updates are below. We continue to experience high error rates with S3 in US-EAST-1, which is impacting various AWS services. We are working hard at repairing S3, believe we understand root cause, and are working on implementing what we believe will remediate the issue.

http://status.aws.amazon.com/

Re:Ask HN: Is S3 down? by Ph4nt0m on Tuesday February 28, 2017 @03:07PM (#13757363)
For some reason, reading about "believe we understand root cause" made me think of: "A distributed system is one in which the failure of a computer you didn't even know existed can render your own computer unusable."
Re:Ask HN: Is S3 down? by perlgeek on Tuesday February 28, 2017 @02:37PM (#13757058)
Maybe you could encourage your colleagues to host the status page outside of AWS?
Re:Ask HN: Is S3 down? by oxguy3 on Tuesday February 28, 2017 @03:02PM (#13757314)
We'll have to wait for the postmortem, but I bet it was an unintentional dependency on S3 that no one realized had come into place until S3 went down -- especially considering how fast they were able to remove the dependency and fix it.
Re:Ask HN: Is S3 down? by koolba on Tuesday February 28, 2017 @03:26PM (#13757509)
This reminds me of a GitHub outage from them having a build dependency on GitHub. IIRC, they tried to roll back to building a prior version but since the site was offline, the build failed.
Re:Ask HN: Is S3 down? by ckozlowski on Tuesday February 28, 2017 @03:09PM (#13757382)
S3 gets used to store a lot of static content. Can't speak for that team, but I'm sure they'll take that feedback. Happy the banner functionality remained unimpeded.
Re:Ask HN: Is S3 down? by all_usernames on Wednesday March 01, 2017 @12:26PM (#13765898)
Possibly AWS status page wisely relied on a third party, which relied on a fourth party, which relied on S3.
Re:Ask HN: Is S3 down? by idlewords on Tuesday February 28, 2017 @03:10PM (#13757390)
It took them ~30 minutes.
Re:Ask HN: Is S3 down? by LunaSea on Tuesday February 28, 2017 @07:56PM (#13759926)
It took them 2 hours actually.
Re:Ask HN: Is S3 down? by johansch on Tuesday February 28, 2017 @02:59PM (#13757277)
Maybe with GCP? :)
Re:Ask HN: Is S3 down? by apapli on Tuesday February 28, 2017 @04:22PM (#13757980)
I'm happy to offer up some spare space on my godaddy hosted Linux plan if that helps...
Re:Ask HN: Is S3 down? by danappelxx on Tuesday February 28, 2017 @02:42PM (#13757107)
Could you go more in depth? What does S3 have anything to do with it?
Re:Ask HN: Is S3 down? by johansch on Tuesday February 28, 2017 @03:20PM (#13757473)
I think the most reasonable guess is that they have some backend system that continously pushes some status json/xml file to an S3 bucket.

Then there's the frontend, that apparently periodically reads this file from S3 and caches the results.

I guess the comment they added on the top after two hours of being in the dark was likely manually added to the web frontend.

Obviously all of this would be hilariously badly designed if it was made this way. Still...

Re:Ask HN: Is S3 down? by unfunco on Tuesday February 28, 2017 @02:53PM (#13757226)
It's where they store the error icon.
Re:Ask HN: Is S3 down? by danappelxx on Wednesday March 01, 2017 @12:15AM (#13761373)
So the "working" icon works, but the "not working" doesn't? I'm not sure that's right.
Re:Ask HN: Is S3 down? by vocatus_gate on Tuesday February 28, 2017 @03:53PM (#13757692)
Tragic comedy gold
Re:Ask HN: Is S3 down? by stretchwithme on Tuesday February 28, 2017 @02:08PM (#13756753)
What we need are status pages that are driven by votes from verified customers, which could also serve to inform the provider about issues.

This would address issues that are only visible from the outside.

Re:Ask HN: Is S3 down? by laughfactory on Tuesday February 28, 2017 @02:39PM (#13757080)
And system monitoring which isn't dependent on itself. Kind of a "duh" kind of thing...
Re:Ask HN: Is S3 down? by ATsch on Tuesday February 28, 2017 @03:55PM (#13757713)
http://outage.report/ does this pretty much, except for the "verified customers" part.
Re:Ask HN: Is S3 down? by j2kun on Tuesday February 28, 2017 @02:07PM (#13756737)
> Amazon, everything is on fire. you are not fooling anyone

Fun story, when I was an intern at Amazon there was actually a warehouse fire. The result was a lot of manual database entry updating as products were determined to be destroyed or still fit for sale.

Re:Ask HN: Is S3 down? by wildmusings on Tuesday February 28, 2017 @02:09PM (#13756773)
I'm curious about what happened to products that were no longer fit for sale, but still fit for use. Do you recall?
Re:Ask HN: Is S3 down? by robaato on Tuesday February 28, 2017 @02:32PM (#13757003)
In the military, a warehouse fire or equivalent suddenly generates a ton of "backdated transfer requests" showing that various stock had been sent to the warehouse just previously!
Re:Ask HN: Is S3 down? by dTal on Tuesday February 28, 2017 @03:44PM (#13757632)
This sounds like rank corruption. Surely such a thing is rare in the military?
Re:Ask HN: Is S3 down? by wildmusings on Tuesday February 28, 2017 @03:49PM (#13757672)
To be fair, there's a plausible explanation for what robaato describes that doesn't involve corruption. Suppose it's standard or common to move things first and then file such "backdated transfer requests". After a fire that destroys everything in a warehouse, there would be a flurry of activity to quickly account for everything that was destroyed, so paperwork that would otherwise have trickled in over a month or two might suddenly be hurriedly filed in a few days.
Re:Ask HN: Is S3 down? by dragonwriter on Tuesday February 28, 2017 @05:26PM (#13758779)
It could just be lackadaisical administration that only gets urgently addressed when there is something perceived as a particular problem.

The military is not exactly known for being great at keeping track of things that aren't nuclear weapons, and sometimes falls short even on those.

Re:Ask HN: Is S3 down? by hobofan on Tuesday February 28, 2017 @02:26PM (#13756952)
There is "Amazon Warehouse Deals", Amazon itself acting as a used products seller on Amazon. This is usually used for returns etc., but I wouldn't be surprised if they also handle something like this.
Re:Ask HN: Is S3 down? by greglindahl on Tuesday February 28, 2017 @06:05PM (#13759120)
There are businesses that specialize in remaindering fire-damaged goods -- mostly stuff that smells like smoke.

They showed up in my town in the early 1980s after one of our local malls had a smokey fire. They sold a bunch of stuff that came from other places, too, including a ton of 15mm miniature soldiers.

Re:Ask HN: Is S3 down? by evtothedev on Tuesday February 28, 2017 @01:24PM (#13756227)
Re:Ask HN: Is S3 down? by fletom on Tuesday February 28, 2017 @01:32PM (#13756316)
I like how this post says "if you look at the AWS Status Page, this what you see". but you can't see the image. because S3 is down.
Re:Ask HN: Is S3 down? by brianpgordon on Tuesday February 28, 2017 @02:41PM (#13757097)
I thought this was funny so I took a screenshot of the blog post and uploaded it to the company Slack. The upload failed because Slack uses S3.

This is getting crazy.

Re:Ask HN: Is S3 down? by kevin_b_er on Tuesday February 28, 2017 @02:08PM (#13756757)
If this isn't good evidence that amazon downright lies on their status page and that no green checkmark should ever be considered trustworthy, I don't know what is.
Re:Ask HN: Is S3 down? by twistedpair on Tuesday February 28, 2017 @02:43PM (#13757129)
Why is the status board hosted on AWS? Most providers host such pages on a 3rd party, specifically for this reason; correlated failure.
Re:Ask HN: Is S3 down? by rinze on Tuesday February 28, 2017 @02:09PM (#13756761)
> edit: in the future, please subscribe to @MyFootballNow for timely AWS service status updates https://pbs.twimg.com/media/C5xdm9_WMAAY7y_.jpg:large

So this is what centralization looks like.

Re:Ask HN: Is S3 down? by skywhopper on Tuesday February 28, 2017 @02:19PM (#13756873)
I hear that their process for updating the status page involves S3.
Re:Ask HN: Is S3 down? by fletom on Tuesday February 28, 2017 @02:29PM (#13756977)
it would appear that you are correct

"The dashboard not changing color is related to S3 issue. See the banner at the top of the dashboard for updates." https://twitter.com/awscloud/status/836656664635846656

Re:Ask HN: Is S3 down? by kevin_b_er on Tuesday February 28, 2017 @03:13PM (#13757414)
I very seriously thought that it was a joke to say that S3 was needed to show the red icon, but apparently they can't update the dashboard about the status of S3 because of S3.
Re:Ask HN: Is S3 down? by knaik94 on Tuesday February 28, 2017 @02:26PM (#13756947)
So... https://twitter.com/awscloud/status/836656664635846656

"The dashboard not changing color is related to S3 issue. See the banner at the top of the dashboard for updates."

Re:Ask HN: Is S3 down? by gtrubetskoy on Tuesday February 28, 2017 @01:59PM (#13756649)
the non-green icon is probably hosted on s3 (i'm not trying to be funny)
Re:Ask HN: Is S3 down? by whafro on Tuesday February 28, 2017 @02:43PM (#13757116)
"Update at 11:35 AM PST: We have now repaired the ability to update the service health dashboard."

Yep.

Re:Ask HN: Is S3 down? by _ao789 on Tuesday February 28, 2017 @02:56PM (#13757254)
I love it, like that fixes the problem! ..now fix the REAL problem
Re:Ask HN: Is S3 down? by artursapek on Tuesday February 28, 2017 @01:36PM (#13756361)
I don't think they intentionally kept the checkmarks there. They probably just didn't update it as quickly as developers made a post on Hacker News (not surprising, they were probably investigating).
Re:Ask HN: Is S3 down? by hobofan on Tuesday February 28, 2017 @01:49PM (#13756519)
After having seen multiple AWS outages/service disruptions, with nothing other than a green checkmark ever showing, I am now very confident that the checkmarks are hardcoded and there is no logic behind them.
Re:Ask HN: Is S3 down? by user5994461 on Tuesday February 28, 2017 @02:01PM (#13756663)
It's already been confirmed by amazon employees on HN that the color can only be changed manually by an employee and it needs a high level of approval.

Also, there are incentives based on colors, so the managers really don't want to admit any failure.

Re:Ask HN: Is S3 down? by bdavisx on Tuesday February 28, 2017 @02:16PM (#13756842)
While this is a personal feeling and I don't have any data (metrics) to back it up: I think a large percentage (and probably a majority) of metrics don't end up helping a company once they are created - especially if any salary or bonuses are based on them. They are always so gamed that they become worthless.

This is a great case in point if true.

Re:Ask HN: Is S3 down? by rab-the-goat on Tuesday February 28, 2017 @03:21PM (#13757478)
Your point hits on a true thing. One problem is that companies measure proxies for performance, not performance itself. A great book on the topic (and related topics) is Weapons of Math Destruction. Anyway, a green checkbox is pretty far into proxie-land. It's not very closely related to client retention or profitability, and now we see it's not even related to operational time of the equipment. Yikes. So a proxy like this is not even worth using as a metric; it can only cause false confidence that some information is known, and that leads to bad decisions. Not the least of which is bonusing incompetent managers.
Re:Ask HN: Is S3 down? by SomeStupidPoint on Tuesday February 28, 2017 @02:32PM (#13756998)
Amazon has metrics so they can tell a story, not so they can measure things.

As a cute example, one of their senior people (in a stats heavy role) couldn't explain how they'd detect if people wanted to be able to automatically order socks and tshirts on a buying cycle outside of what I call the "scheduling horizon", eg every 3-6mos. (Things I need regularly, but sparsely enough it doesn't stand out to do proactively -- eg, I buy socks when they all have holes, not on a reasonable replacement cycle.)

Re:Ask HN: Is S3 down? by KnoopKnoop on Tuesday February 28, 2017 @02:12PM (#13756799)
Yup probably some incentives due to SLA's for their larger customers.
Re:Ask HN: Is S3 down? by mschuster91 on Tuesday February 28, 2017 @02:13PM (#13756817)
> Also, there are incentives based on colors, so the managers really don't want to admit any failure.

A textbook case of "wrong incentives". #1 incentive should be satisfied customers.

Re:Ask HN: Is S3 down? by TeMPOraL on Tuesday February 28, 2017 @04:57PM (#13758457)
You can have a high level of customer satisfaction if you lie to them and arrange so that they don't even notice, and by having a good damage control strategy for when some customers do notice they aren't getting that they were promised.

Such approach has better ROI than actually doing high quality products or services, which is why so much of what we buy is utter shit. That's especially true on the mass market, when satisfaction of individual customers doesn't impact your company at all, as long as they're not complaining too loud.

Re:Ask HN: Is S3 down? by oneeyedpigeon on Tuesday February 28, 2017 @02:17PM (#13756853)
Are any incentives affected by customer complaints? Because if so, all we need to do is complain and they might actually start using the status system meaningfully. (I'm sure this is wilful thinking, though; I doubt people don't complain in this situation!)
Re:Ask HN: Is S3 down? by fletom on Tuesday February 28, 2017 @01:45PM (#13756470)
when you start investigating an outage, that is exactly when you should change your checkmark to yellow if not red. if you're as big as AWS there should not be any more than a minute or two between when your service goes down and when you actually update your status page to show that.
Re:Ask HN: Is S3 down? by CaptSpify on Tuesday February 28, 2017 @03:08PM (#13757374)
It should actually just be automated.