throwaway985325

I work on an internal service, and my weekly metrics reflect the health of a lot of services. It's more often than not that those metrics look like a printout from a seismograph sitting on an active fault.

I don't know if other cloud providers were impacted, but I'm willing to bet they were not impacted intermittently for 6 hours.

Don't use AWS, or at least don't put all your eggs into the AWS basket.

AWS has gotten to the service and infrastructure footprint it has today by death marching mediocre talent to ship untested and brittle things. This is true all the way down into the physical infrastructure, which often breaks and causes regional havoc. Great examples from the last 6 months include the 40 minute outage in US-WEST-2 in September, or the several hour event in US-EAST-1 during the East Coast wind storms. Bubble gum and baling wire.

They're trying to stabilize things, but leadership and sales are often committing to large spend customers and financing new features with technical debt (when current technical debt is not being serviced). Those at the top are not really concerned with running a solid or innovative platform as much as just pure growth and revenue.