Thanks for the comment and I love all the insight but serious question about:

> there are going to be a lot of unhappy people cancelling their weekend plans right now.

The week just started... are you saying that you can already anticipate the war room for an event like this to extend through this coming weekend?

Haha, no, oops. My last few days have been a bit crazy and I'm on holiday, so I just thought it was Sunday ;)

Hehe. Reminds me of my Murphy's law momemts. Whenever I am on a holiday, a deployment will fail, and I will have to login and check while my family plays volleyball on the beach.

I'd tell your boss to hire someone to replace you. What if instead of on holiday, you were dead? Kinda hard to login and fix their shit when you're dead.

I agree. I quit every job where they don't respect my personal time, especially vacation. Life is just too short. It's bad enough what I've been through with companies and their stingy PTO policies (US).

That sounds like a very low bus factor to me.

I was smoked up in colorado for the old "Lets take down Dyn real quick" BS a couple years ago. I know this very well, things like to run for months without issue, and I'm off for an afternoon with the phone ringing.

It's a good holiday if you can no longer remember what day it is.

But not good enough if you’re checking HN ;)

I'm on holiday in Japan at the moment and have no idea what day it is. Only checking HN as I'm currently in Osaka waiting for the Shinkansen to Hiroshima. ;)

Do you seriously believe reading something like HN is work for its viewers??

I think they meant that if you're on vacation you maybe shouldn't be looking at the computer.

my memorial weekend on a few acre olympic rain forest camping trip: regular signal hunting for hn. Oddly found a 6x6 area which after a reboot with provide lte. after idle would have to reboot again to open some more links

My guess was either that

OR there wasn't backpressure on a cascading failover so as services failed they increasingly failed to more and more overloaded systems

OR there WAS backpressure and it was the luck of the draw whether you were queued into an error page or got good data

OR the autoscaling couldn't keep up with the onsale window. This used to happen in ticketing a lot. Ticketmaster has a talk somewhere where they talk about warming the scaling load and server cache in anticipation of big ticketing onsales. The time it took to autoscale was just too long.

The systems at Amazon are too large to leverage autoscaling. When I was there (in Marketplace) it was generally estimated that Amazon used about 80X the capacity of their entire AWS public cloud.

I find that implausible. I know they run a huge infrastructure but why would they be using 80X of their AWS public cloud? Thousands of companies if not millions at this stage use AWS and some of them are not insignificant in size.

WTF? 80 times the capacity of their entire AWS infrastructure?? That seems insane.

No, the exact quote was:

> Amazon used about 80X the capacity of their entire AWS public cloud

which is probably closer to "the capacity that is available via AWS is a tiny, tiny fraction of their overall computing power.. therefore adding it back in when things are falling over doesn't actually solve any problems."

I don't understand the difference between the wording in your post and the post you're replying to.

I think they are using the term 'capacity' to mean 'spare capacity'. I.e. that Amazon's entire compute usage is 80x the spare capacity, so scaling even a small amount would consume any spare capacity in AWS. Still, it seems hard to believe.

I interpreted it to mean:

1. he misspoke and meant "80%" of the AWS capacity, which I agree seems implausible. 2. Amazon does not run on AWS because Amazon is 80x more than all of AWS infrastructure. This also seems implausible because of Netflix. In fact, there's an article out there that said AWS exceeded Amazon's capacity within 1 quarter!

I still don't understand what that has to do with autoscaling exactly

I did’t understand either. I suspect it’s the semantic difference between ‘public cloud’ and ‘infrastructure’. I don’t know what that difference is really.

When I checked it out, the site seemed responsive but search was broken. I searched "usb flash drive" and was presented with 2 results. I pondered this for a moment and then realized that there were 400 pages with 2 results/page (as far as I checked.) Perhaps there is some load shedding algorithm that reduces the result count per page to reduce load. It did discourage me from further searching so I guess it worked. ;)

smile.amazon.com was working fine during the outage, if that helps narrow it down...

It wasn't though. I saw an item being listed as a Prime Day deal on the normal Amazon site, then I searched for it through smile.amazon.com and it wasn't there. Went back to non-smile and it was there (but throwing errors so I couldn't click it anyway...)

I'm seeing this also. Same item is $169 on smile.amazon.com, versus $79 on www. Neither is working properly though.

Not for me: a page that gave a fast 503 on amazon.com would spin indefinitely on smile.amazon.com

Weekend plans? Prime Day is going to be over in the early part of the week.

Does Amazon expect a fix to be deployed ASAP after the immediate crisis is averted?

My bet is they maxed out on the SQS FIFO 300 messages per second limit.

I have no idea really (I stopped working on the retail websites in 2013) but my gut feel is that that's about a couple of orders of magnitude too low at least.

But, yes, unforeseen rate limits and size limits can cause many hilarious things to happen. I've seen a few good ones in my time. In particular, when somebody sets an upper size on an in-memory table and commits it thinking "Well, I added a few orders of magnitude safety margin - that should be enough for anybody", that's probably going to become an incident at some point in the distant future ;) With luck, the throttling or failure behaviour will only affect a few people, and it'll be spotted by looking at traffic graphs and noticing a very slightly elevated rate of service errors. If you're unlucky, though, when you hit the limit the whole service slows down, locks up, or just plain crashes, and something like this happens...

Yup, many an outage has something like this at its core. At my company, to address that, we've built an internal library for enforcing rate limits and size limits that is (a) configurable on the fly and (b) generates logs, so that we can trigger an alert whenever any limit reaches 70% of capacity. And thus, hopefully, head these things off before they reach the tipping point.

https://blog.scalyr.com/2014/08/99-99-uptime-9-5-schedule/

Don't forget to alert when the double derivative of capacity growth is above a (very low!) threshold, that can catch an explosive problem far earlier than the 70% mark, by which time it might be growing at 5% per second.

It might also be a momentary blip that you really shouldn't need to wake a human up for. The hard part of monitoring imo is that perfect balance. Asymtotically approaching nirvana but never quite reaching.

All monitoring has that problem :-) But if your rate of rate of change has been positive for some suitable amount of time for the context, it's worth waking a human up over it, because the amount of resource remaining is dwindling exponentially.

That sounds like a very interesting metric to track and report on. Do you have any further references that discuss that approach?

Not offhand, sorry - I first encountered the notion at a conference some years back, but I've long since gotten out of operations so it's all a bit dusty for me now.

That's cool, but it's a bit more complicated then rate limits.

I once owned an EAV DB for storing small config values. Someone wrote a great wrapper around it that made it seem like a proper DB with many tables. Since it was for storing configs this library cached the whole thing in memory on startup. Zoom to 5 years later, we have 10+ settings for every customer in this store, and one day all our hosts keep failing over. As it turns out that small table for configs was north of 5 gigs and destroying our heap.

This is their official rate limit for the FIFO type SQS. Although I bet the Amazon folks know a couple people around the office who can up that limit :)

oncall is the worst

Not oncall for Amazon but I don't particularly mind it. It sucks to have to be near a computer for the time, but I actually find digging in and finding that needle in a haystack pretty fun. On a good team, the life interruption is minimal.

As much as I like digging,debugging and learning that comes with it, fixing things under tremendous time pressure wears me out :(

When I was on-call for a big project, I jumped every time my phone rang. It wasn't a good experience at all.

"We are at the fair, I have 18 customers lined up but I can't take orders. Can you fix this now or should I start taking them on paper?"

You know the joke that atheists don't believe in god until the plane nosedives? That was the moment for me :)

"Oh, I... I need to see the error message before I can say anything. We don't have any errors logged. Hmm... Would you allow me to remote into your device?"

"Can't do that. We lost the internet connection here, sorry."

Ugh.

How long have you been on call for and how many 3am wakeup calls have you had? I used to think these same. Protip: you won't always be on a good team.

I'm in Asia and always thought I'd make a great on call person for those of you who haven't jumped ship. I should apply somewhere.

Hah. I'm genuinely surprised this isn't more of a thing.

We reduced the stress of on-call by introducing processes that supported creativity and reduced panic:

https://zwischenzugs.com/2017/04/04/things-i-learned-managin...

I prefer to be oncall as long as I'm empowered to prioritize software improvements that reduce ops burden and improve customer experience.