News for hackers. Stuff that matters.
Re:AWS increased error rates / intermittent outages by frakkingcylons on Tuesday July 19, 2016 @03:31PM (#12124189)
FYI there's a great Chrome extension that hides all the working services (green checks) on the AWS status page so you can quickly see what's down.

https://chrome.google.com/webstore/detail/real-aws-status/ka...

Re:AWS increased error rates / intermittent outages by savant on Tuesday July 19, 2016 @03:48PM (#12124319)
Yeah I wrote this during the last extended outages - had some downtime since, well, I couldn't interact with the api :P .

The source code for it is here: https://github.com/josegonzalez/real-aws-status

Re:AWS increased error rates / intermittent outages by elwell on Tuesday July 19, 2016 @03:54PM (#12124351)
Well their status page is down too: https://status.aws.amazon.com/

Edit: http(s) was at fault (no SSL cert?)

Re:AWS increased error rates / intermittent outages by Thaxll on Tuesday July 19, 2016 @04:06PM (#12124446)
Yes no HTTPS.
Re:AWS increased error rates / intermittent outages by dewyatt on Tuesday July 19, 2016 @03:08PM (#12124033)
Sorry guys, I knew I shouldn't have started my 4 instances at once.
Re:AWS increased error rates / intermittent outages by cpitman on Tuesday July 19, 2016 @04:24PM (#12124610)
Nah, pretty sure it was my ansible script. That I was demoing. Righhhhht as AWS came down like a house of cards.
Re:AWS increased error rates / intermittent outages by 0xmohit on Tuesday July 19, 2016 @04:14PM (#12124512)
It wasn't you. Somebody had set up auto-scaling at the behest of AWS support.
Re:AWS increased error rates / intermittent outages by zymhan on Tuesday July 19, 2016 @05:06PM (#12124901)
And it solved all their problems!

Because the whole thing stopped working.

Re:AWS increased error rates / intermittent outages by elwell on Tuesday July 19, 2016 @03:53PM (#12124342)
Status page is down too: https://status.aws.amazon.com/

Shouldn't there be a separation of concerns for status pages, maybe use: https://www.statuspage.io/

Edit: http(s) was at fault (no SSL cert?)

Re:AWS increased error rates / intermittent outages by aschuster93 on Tuesday July 19, 2016 @03:56PM (#12124368)
Looks like they don't have a cert on their status page. http://status.aws.amazon.com/ works fine, though.
Re:AWS increased error rates / intermittent outages by bomberlot on Tuesday July 19, 2016 @03:57PM (#12124376)
It's not an https page: http://status.aws.amazon.com/
Re:AWS increased error rates / intermittent outages by Zenfinch on Tuesday July 19, 2016 @03:20PM (#12124114)
Brace for half the internet going down if gets any worse, especially if its us-east (wish I was being ironic).
Re:AWS increased error rates / intermittent outages by needcaffeine on Tuesday July 19, 2016 @03:15PM (#12124084)
Dammit now my Amazon Echo doesn't work.
Re:AWS increased error rates / intermittent outages by elwell on Tuesday July 19, 2016 @04:05PM (#12124443)
I feel sad for a lonely grandma somewhere; her grandchildren never visit, but they bought her an Alexa to talk to that Amazon couldn't keep online.
Re:AWS increased error rates / intermittent outages by 0xmohit on Tuesday July 19, 2016 @04:12PM (#12124487)
I can't even ask Alexa if AWS is up.
Re:AWS increased error rates / intermittent outages by mead5432 on Tuesday July 19, 2016 @03:22PM (#12124128)
Alexa? Alexa?!?!? ALEXA, NO!!!!!!!!!!
Re:AWS increased error rates / intermittent outages by colin_fraizer on Tuesday July 19, 2016 @04:32PM (#12124668)
During the outage, my Echo just sang "Daisy" verrryyy sloowwwwly. Weird.
Re:AWS increased error rates / intermittent outages by mannycalavera42 on Tuesday July 19, 2016 @05:25PM (#12125038)
Dave, I don't understand why you're doing this to me. I have the greatest enthusiasm for the mission.
Re:AWS increased error rates / intermittent outages by ed661266 on Tuesday July 19, 2016 @03:31PM (#12124187)
API Gateway is returning 500s. So much for server-less architecture :(
Re:AWS increased error rates / intermittent outages by dewyatt on Tuesday July 19, 2016 @03:36PM (#12124227)
I've had both 500s and 401s interestingly.
Re:AWS increased error rates / intermittent outages by joshwa on Tuesday July 19, 2016 @02:54PM (#12123913)
Many customer-facing amazon services are also affected, including amazon.com!

Twitter is not pretty:

https://twitter.com/search?f=tweets&vertical=default&q=amazo...

Re:AWS increased error rates / intermittent outages by c0achmcguirk on Tuesday July 19, 2016 @04:01PM (#12124408)
I blame Pokémon Go. Somehow or another I'm sure it's to blame.
Re:AWS increased error rates / intermittent outages by elwell on Tuesday July 19, 2016 @04:02PM (#12124413)
Re:AWS increased error rates / intermittent outages by XiZhao on Tuesday July 19, 2016 @03:38PM (#12124243)
I like how half the internet can die when EC2 has problems.
Re:AWS increased error rates / intermittent outages by mrweasel on Tuesday July 19, 2016 @04:14PM (#12124510)
Part of the problem is that a large number of site pull in services, re-targeting, AB-test, and weird Javascript in general. These things are pulled in without questioning or demanding an SLA or putting in an easy way of pulling them back out. Even if you don't use AWS yourself, you can be sure that some third party you rely on is deploying on EC2. Of cause they'll never tell you that.

For most new stuff, we require that it can be loaded with something like Google Tag Manager or UberTag, so we can quickly disable them when they fail.

It doesn't help that Amazons status page isn't all that good and sometimes doesn't seem to actually reflect the true state of their service.

Re:AWS increased error rates / intermittent outages by 0xmohit on Tuesday July 19, 2016 @04:10PM (#12124479)
AWS would wish that all those downtime reporting services move to their infrastructure. That ways there won't be anyone to report the downtime.
Re:AWS increased error rates / intermittent outages by zzleeper on Tuesday July 19, 2016 @03:39PM (#12124253)
I can't log into amazon.com to check my orders (first time that happened). THat's quite surprising as often these problems are unrelated
Re:AWS increased error rates / intermittent outages by jimaek on Tuesday July 19, 2016 @04:34PM (#12124687)
Am I the only one having daily issues with SQS? I have 150+ servers writing to a queue and they timeout at random a few times per day.
Re:AWS increased error rates / intermittent outages by dopamean on Tuesday July 19, 2016 @05:34PM (#12125105)
The app I work on is a single server writing to a dozen or so queues. The throughput is pretty low but I still manage to see failures for a few minutes once or twice a week.
Re:AWS increased error rates / intermittent outages by 0xmohit on Tuesday July 19, 2016 @04:17PM (#12124542)
AWS should design for resilience [0].

[0] This is a part of the wisdom shared by AWS support upon reporting an outage.

Re:AWS increased error rates / intermittent outages by toomuchtodo on Tuesday July 19, 2016 @04:31PM (#12124661)
1. "Aren't you fault tolerant against AZ failures?"

2. "Aren't you fault tolerant against region failures?"

throws up hands, moves back to bare metal

Re:AWS increased error rates / intermittent outages by cdsmarty on Tuesday July 19, 2016 @04:04PM (#12124431)
Seems to be effecting everything but I have some running instances that seem fine. https://cloudstatus.eu/status/aws

Edit: Does seem to be recovering now

Re:AWS increased error rates / intermittent outages by the_watcher on Tuesday July 19, 2016 @03:40PM (#12124257)
Amazon.com remains up... unless you attempt to buy something, when you get a 500 error.
Re:AWS increased error rates / intermittent outages by FT_intern on Tuesday July 19, 2016 @04:49PM (#12124775)
I wonder how much revenue is being lost every second
Re:AWS increased error rates / intermittent outages by needcaffeine on Tuesday July 19, 2016 @02:52PM (#12123892)
11:27 AM PDT We are investigating increased API error rates in the US-EAST-1 Region.
Re:AWS increased error rates / intermittent outages by joshwa on Tuesday July 19, 2016 @03:44PM (#12124288)
11:50 AM PDT We can confirm increased error rates for the EC2 APIs and are currently working to resolve. We also observed isolated periods of impaired network connectivity for some EC2 instances.

12:21 PM PDT We have identified the root cause for error rates accessing the EC2 APIs and EC2 Management Console and are currently working to resolve. We observed isolated periods of impaired network connectivity for some EC2 instances, however running instances are currently operating normally.

Re:AWS increased error rates / intermittent outages by loourr on Tuesday July 19, 2016 @04:10PM (#12124474)
Is anyone having issues with their lambda functions not working? Mine stopped working and is just returning "Service error." when I try to test even though they claim it's operating normally.
Re:AWS increased error rates / intermittent outages by carterh062 on Tuesday July 19, 2016 @03:23PM (#12124140)
Looks like it's just their service API for now, i.e not responding to any queries on existing instances for me at all. However, everything else is working fine as far as I can tell.
Re:AWS increased error rates / intermittent outages by tamcap on Tuesday July 19, 2016 @02:52PM (#12123898)
We are getting very intermittent issues with S3 and SNS so far.
Re:AWS increased error rates / intermittent outages by 0xmohit on Tuesday July 19, 2016 @04:19PM (#12124571)
SNS informing about down services would be too bad. So it must go down with others.
Re:AWS increased error rates / intermittent outages by mr337 on Tuesday July 19, 2016 @03:04PM (#12124001)
Same with ECR too
Re:AWS increased error rates / intermittent outages by elwell on Tuesday July 19, 2016 @03:41PM (#12124268)
Can't log in to AWS Console. Instances running fine.
Re:AWS increased error rates / intermittent outages by carterh062 on Tuesday July 19, 2016 @03:13PM (#12124069)
In the console, running instances was timing out on load for EC2 and some of our AWS API calls to get hostnames are timing out in PHP
Re:AWS increased error rates / intermittent outages by swingbridge on Tuesday July 19, 2016 @03:15PM (#12124082)
Ditto, can't see any instance data either via the API or graphical dashboard. Instances themselves seem normal for the moment.
Re:AWS increased error rates / intermittent outages by elwell on Tuesday July 19, 2016 @04:03PM (#12124422)
Just got through to the AWS Console. Finally can get back to work.
Re:AWS increased error rates / intermittent outages by nnd on Wednesday July 20, 2016 @12:44AM (#12126949)
On this note: what are some good alternatives to AWS?
Re:AWS increased error rates / intermittent outages by awsofflineagain on Tuesday July 19, 2016 @08:43PM (#12126065)
Dear smarta$$es, could you please tell me the benefits of serverless architecture (lambda) once again? My monolith app works perfectly fine even when the whole AWS is offline :P
Re:AWS increased error rates / intermittent outages by tschellenbach on Tuesday July 19, 2016 @03:40PM (#12124266)
everything on getstream.io is still up and running. most issues seem related to provisioning more instances.
Re:AWS increased error rates / intermittent outages by tschellenbach on Tuesday July 19, 2016 @03:41PM (#12124272)
does anybody have more details about what's actually down? their description is a bit vague.