News for hackers. Stuff that matters.
Re:Ask HN: Is S3 down? by cperciva on Tuesday February 28, 2017 @05:05PM (#13758546)
S3 is currently (22:00 UTC) back up.

The timeline, as observed by Tarsnap:

    First InternalError response from S3: 17:37:29
    Last successful request: 17:37:32
    S3 switches from 100% InternalError responses to 503 responses: 17:37:56
    S3 switches from 503 responses back to InternalError responses: 20:34:36
    First successful request: 20:35:50
    Most GET requests succeeding: ~21:03
    Most PUT requests succeeding: ~21:52
Re:Ask HN: Is S3 down? by josephb on Tuesday February 28, 2017 @05:31PM (#13758828)
Thanks for taking the time to post a timeline from the perspective of an S3 customer. It will be interesting to see how this lines up against other customer timelines, or the AWS RFO.
Re:Ask HN: Is S3 down? by kaishiro on Tuesday February 28, 2017 @09:48PM (#13760531)
Playing the role of the front-ender who pretends to be full-stack if the money is right, can someone explain the switch from internal error to 503 and back? Is that just them pulling s3 down while they investigate?
Re:Ask HN: Is S3 down? by cperciva on Wednesday March 01, 2017 @02:16AM (#13761932)
My guess based on the behaviour I've seen is that internal nodes were failing, and the 503 responses started because front-end nodes didn't have any back-end nodes which were marked as "not failing and ready for more requests". When Amazon fixed nodes, they would have marked the nodes as "not failed", at which point the front ends would have reverted to "we have nodes we can send traffic to" behaviour.
Re:Ask HN: Is S3 down? by greenleafjacob on Tuesday February 28, 2017 @11:16PM (#13761057)
Could be anything. Most likely scenario is the internal error is a load shedding error and the 503s were when the system became completely unresponsive. If it was a configuration issue then it is more likely that it would have directly recovered rather than going 'internal error -> 503 -> internal error'.
Re:Ask HN: Is S3 down? by hmottestad on Wednesday March 01, 2017 @02:12AM (#13761908)
503 is typically what we see when our proxy can't connect to the backend server. We usually get 500 with internal server error when we've messed up the backend server.

So it's likely that the first 500s were the backend for s3 failing, then they took the failing backends offline causing the load balancers to throw 503 because they couldn't connect to the backend.

Re:Ask HN: Is S3 down? by Twirrim on Wednesday March 01, 2017 @11:52AM (#13765431)
S3 is not a monolithic architecture, Amazon is a strong proponent of Service Oriented Architecture for producing scalable platforms.

There are a number of services behind the front end fleet in S3's architecture that handle different aspects of returning a response. Each of those will have their own code paths in the front end, very likely developed by different engineers over the years. As ever, appropriate status codes for various circumstances are something that always seems to spur debate amongst developers.

The change in status code would likely be a reflection of the various components entering unhealthy & healthy states, triggering different code paths for the front end... which suggests whatever happened might have had quite a broad impact, at least on their synchronous path components.

[dead]
Re:Ask HN: Is S3 down? by endersshadow on Tuesday February 28, 2017 @05:22PM (#13758742)
Soundcloud recovering from this failure and S3 being operational are two separate issues. We use S3 and it will take us nominally an hour to recover after S3 went up, for example.

S3 has started working as of about 20 minutes ago, and things are running smoothly.

Re:Ask HN: Is S3 down? by quakeguy on Tuesday February 28, 2017 @05:52PM (#13759003)
Thanks!
Re:Ask HN: Is S3 down? by jeffasinger on Tuesday February 28, 2017 @05:57PM (#13759052)
There are other Amazon services that were affected. For example, we're still not seeing auto scaling groups working correctly.
Re:Ask HN: Is S3 down? by ta_wh on Tuesday February 28, 2017 @05:21PM (#13758734)
"[RESOLVED] Increased Error Rates

Update at 2:08 PM PST: As of 1:49 PM PST, we are fully recovered for operations for adding new objects in S3, which was our last operation showing a high error rate. The Amazon S3 service is operating normally."

https://status.aws.amazon.com/

Re:Ask HN: Is S3 down? by thenewregiment2 on Tuesday February 28, 2017 @05:25PM (#13758764)
oh the famous downvote for the smear campaign. just admit it.
Re:Ask HN: Is S3 down? by joatmon-snoo on Tuesday February 28, 2017 @05:31PM (#13758835)
You're getting downvotes because you don't understand that B being down is not an effective indicator of the status of A, even if B depends on A.
Re:Ask HN: Is S3 down? by ta_wh on Tuesday February 28, 2017 @05:30PM (#13758822)
Think you're mistaken, I don't have downvote privileges!
Re:Ask HN: Is S3 down? by espeed on Tuesday February 28, 2017 @05:59PM (#13759062)
Claiming a statement is false when it's demonstrably true is something that will likely get downvoted every time. It's misleading to others and fills the board with noise.