News for hackers. Stuff that matters.
Re:Amazon packages pile up after AWS outage spawns delivery ... by riffic on Tuesday December 07, 2021 @09:16PM (#29480582)
do they not teach people what a failure domain is anymore?

https://en.wikipedia.org/wiki/Failure_domain

Clouds aren't magic. They require a certain amount of operational confidence in order to understand that, yes, an entire region can fall out from under you at any time and it's your responsibility to detect and deploy into an unaffected region if possible.

edit: Generally, one entire region will not fail. However, core services like STS rely on us-east-1 so it's particularly susceptible to disruption.

Re:Amazon packages pile up after AWS outage spawns delivery ... by Spooky23 on Tuesday December 07, 2021 @10:14PM (#29480902)
People get slagged on for not having inter-region redundancy. But unless your business model can accommodate that, you’re introducing another failure domain and alot of money for duplicative infrastructure, network fees, etc.

For many use cases, it’s acceptable to shrug and blame AWS for a failure. It’s harder when your high availability solution fails independently, which they almost always do more than US-East-1

Re:Amazon packages pile up after AWS outage spawns delivery ... by phamilton on Wednesday December 08, 2021 @01:01AM (#29481725)
If your site goes down while everyone else's goes down, it's weirdly forgivable.

And being the only one up doesn't win as many market cred points and you'd think.

Re:Amazon packages pile up after AWS outage spawns delivery ... by Spooky23 on Wednesday December 08, 2021 @08:29AM (#29484436)
Exactly. Often people want the perception of reliability.

For many services, it makes more sense to make it reliable than not. For other services, it makes more sense to think about the engineering of the solution in the field.

Example: McDonald’s product images on kiosks are apparently in S3 and not cached locally. Seems like a dumb idea to me, but I wouldn’t try to build a more reliable cloud storage backend to control that risk.

Re:Amazon packages pile up after AWS outage spawns delivery ... by deathanatos on Tuesday December 07, 2021 @10:00PM (#29480820)
> do they not teach people what a failure domain is anymore?

Empirically? No.

But seriously, instead of making every dev team consuming an AWS product hire the extra engineers required to build a system that spans multiple failure domains — which lets be honest, companies won't — why hasn't Amazon just hired the engineers required to do it for me?

> Generally, one entire region will not fail.

Laughs in global failures.

Re:Amazon packages pile up after AWS outage spawns delivery ... by sithadmin on Tuesday December 07, 2021 @09:54PM (#29480781)
Honestly, no. Not frequently enough.

There are also way too many successful, public cloud-native businesses running without any semblance of a Business Continuity or DR Plan among any of their teams. I wish I could name and shame some of the more egregious cases I have seen.

Re:Amazon packages pile up after AWS outage spawns delivery ... by donavanm on Wednesday December 08, 2021 @02:05AM (#29481999)
> However, core services like STS rely on us-east-1 so it's particularly susceptible to disruption.

FYI this example hasn’t been true for a while. STS regional endpoints are generally what you should be using these days. The “global” us-east-1 endpoint still works, and may be the default for some clients, but isn’t a requirement.

Re:Amazon packages pile up after AWS outage spawns delivery ... by RyJones on Tuesday December 07, 2021 @09:36PM (#29480683)
In my experience, once an item is done done done, it’s out of your hands.