News for hackers. Stuff that matters.
Re:AWS us-east-1 outage by dylan604 on Tuesday December 07, 2021 @03:03PM (#29477502)
The fun thing about these types of outages are seeing all of the people that depend upon these services with no graceful fallback. My roomba app will not even launch because of the AWS outage. I understand that the app gets "updates" from the cloud. In this case "updates" is usually promotional crap, but whatevs. However, for this to prevent the app launching in a manner that I can control my local device is total BS. If you can't connect to the cloud, fail, move on and load the app so that local things are allowed to work.

I'm guessing other IoT things suffer from this same short sitedness as well.

Re:AWS us-east-1 outage by lordnacho on Tuesday December 07, 2021 @03:52PM (#29478061)
If you did that some clever person would set up their PiHole so that their device just always worked, and then you couldn't send them ads and surveil them. They'd tell their friends and then everyone would just use their local devices locally. Totally irresponsible what you're suggesting.
Re:AWS us-east-1 outage by apexalpha on Tuesday December 07, 2021 @03:59PM (#29478155)
A little off-topic, but there are people working on it: https://valetudo.cloud/

It's a little harder than blocking the DNS unfortunately. But nonetheless it always brings a smile to my face to see that there's a FOSS frontier for everything.

Re:AWS us-east-1 outage by beamatronic on Tuesday December 07, 2021 @04:02PM (#29478180)
An even more clever person would package up this box, and sell it, along with a companion subscription service, to help busy folks like myself.
Re:AWS us-east-1 outage by dylan604 on Tuesday December 07, 2021 @04:12PM (#29478289)
But this new little box would then be required to connect to the home server to receive updates. Guess what? No updates, no worky!! It's a vicious circle!!! Outages all the way down
Re:AWS us-east-1 outage by arksingrad on Tuesday December 07, 2021 @04:49PM (#29478642)
this is why everyone runs piholes and no one sees ads on the internet anymore, which killed the internet ad industry
Re:AWS us-east-1 outage by dylan604 on Tuesday December 07, 2021 @05:09PM (#29478846)
Dear person from the future, can you give me a hint on who wins the upcoming sporting events? I'm asking for a friend of course
Re:AWS us-east-1 outage by 0des on Tuesday December 07, 2021 @05:26PM (#29479012)
Also, what's the verdict on John Titor?
Re:AWS us-east-1 outage by s_dev on Tuesday December 07, 2021 @03:10PM (#29477584)
>The fun thing about these types of outages are seeing all of the people that depend upon these services with no graceful fallback.

Whats a graceful fallback? Switching to another hosting service when AWS goes down? Wouldn't that present another set of complications for a very small edge case at huge cost?

Re:AWS us-east-1 outage by rehevkor5 on Tuesday December 07, 2021 @03:41PM (#29477940)
Usually this refers to falling back to a different region in AWS. It's typical for systems to be deployed in multiple regions due to latency concerns, but it's also important for resiliency. What you call "a very small edge case" is occurring as we speak, and if you're vulnerable to it you could be losing millions of dollars.
Re:AWS us-east-1 outage by simmanian on Tuesday December 07, 2021 @03:58PM (#29478138)
AWS itself has a huge single point of failure on us-east-1 region. Usually, if us-east-1 goes down, others soon follow. At that point, it doesn't matter how many regions you're deploying to.
Re:AWS us-east-1 outage by scoopertrooper on Tuesday December 07, 2021 @04:12PM (#29478296)
My workloads on Sydney and London are unaffected. I can't speak for anywhere else.
Re:AWS us-east-1 outage by lowbloodsugar on Tuesday December 07, 2021 @04:15PM (#29478320)
"Usually"? When has that ever happened?
Re:AWS us-east-1 outage by dylan604 on Tuesday December 07, 2021 @05:11PM (#29478866)
Re:AWS us-east-1 outage by lowbloodsugar on Wednesday December 08, 2021 @12:05AM (#29481443)
Thanks. That shows that OPs claim that "Usually, if us-east-1 goes down, others soon follow." is false.
Re:AWS us-east-1 outage by dijit on Wednesday December 08, 2021 @02:39AM (#29482184)
Cognito (and r53) have hard dependencies in us-east-1. It’s mentioned up and down this thread.
Re:AWS us-east-1 outage by lowbloodsugar on Wednesday December 08, 2021 @12:07PM (#29487327)
I don't want to minimize the impact from cognito and r53, but that's quite a different scale of failure than the OP implies. It's been a while since I used AWS, but we had multiple regions and saw no impact to our services in other regions the one time that us-east-1 had a major failure. And we used r53.
Re:AWS us-east-1 outage by simmanian on Wednesday December 08, 2021 @12:24PM (#29487604)
Perhaps I could've been more precise with my words. What I meant to say is IAM and r53 are two of many critical services that all depend on us-east-1. It goes without saying that if those services go down in us-east-1, the whole AWS is affected. This doesn't just happen "usually." When IAM goes down, AWS experiences major issues across all regions. If you were okay, perhaps you got lucky? Our team has 7 different prod regions and we see multiple regions go down every time a problem of this scale occurs.

If your product requires 100% uptime, you may need to look at backup options or design your product in such a way that can handle temporary cloud failures.

Re:AWS us-east-1 outage by cowmoo728 on Tuesday December 07, 2021 @06:59PM (#29479782)
I heard from someone at <big media company> that they couldn't switch to their fallback region because they couldn't update DNS on Route53. All the console commands and web interface were failing.
Re:AWS us-east-1 outage by stevehawk on Tuesday December 07, 2021 @04:06PM (#29478225)
probably not possible for a lot mroe services than you'd think because AWS Cognito has no decent failover method
Re:AWS us-east-1 outage by twistedpair on Tuesday December 07, 2021 @08:03PM (#29480203)
There is a company that delivers broadcast video ads to hundreds of TV stations on demand. The ad has to run and run now, so they cannot tolerate failure.

They write the videos to GCS storage in Google Cloud, and to S3 in AWS. Every point of their workflows are checkpointed and cross referenced across GCP and AWS. If either side drops the ball, the other picks it up.

So yes, you can design a super fault tolerant system. This company did it because failing to deliver a few ads would mean lose of major contracts.

Re:AWS us-east-1 outage by winrid on Tuesday December 07, 2021 @03:21PM (#29477693)
In this case, just connect over LAN.
Re:AWS us-east-1 outage by s_dev on Tuesday December 07, 2021 @03:26PM (#29477754)
Right -- I think I've misread OP as graceful fallback e.g. working offline.

Rather than implement a dynamically switching backup in the event of AWS going down which is not trivial.

Re:AWS us-east-1 outage by politician on Tuesday December 07, 2021 @03:22PM (#29477703)
Or BlueTooth.
Re:AWS us-east-1 outage by winrid on Tuesday December 14, 2021 @02:16AM (#29549105)
Well, BT is a hell of a protocol. I wouldn't wish that on anyone.
Re:AWS us-east-1 outage by itisit on Tuesday December 07, 2021 @03:33PM (#29477838)
> Wouldn't that present another set of complications for a very small edge case at huge cost?

One has to crunch the numbers. What does a service outage cost your business every minute/hour/day/etc in terms of lost revenue, reputational damage, violated SLAs, and other factors? For some enterprises, it's well worth the added expense and trouble of having multi-site active-active setups that span clouds and on-prem.

Re:AWS us-east-1 outage by sneak on Tuesday December 07, 2021 @04:15PM (#29478321)
Now think of how many assets of various governments' militaries are discreetly employed as normal operational staff by FAAMG in the USA and have access to cause such events from scratch. I would imagine that the US IC (CIA/NSA) already does some free consulting for these giant companies to this end, because they are invested in that Not Being Possible (indeed, it's their job).

There is a societal resilience benefit to not having unnecessary cloud dependencies beyond the privacy stuff. It makes your society and economy more robust if you can continue in the face of remote failures/errors.

It is December 7th, after all.

Re:AWS us-east-1 outage by vdqtp3 on Tuesday December 07, 2021 @06:48PM (#29479677)
> I would imagine that the US IC (CIA/NSA) already does some free consulting for these giant companies

This comment is how I know you don't work in the public sector. Those agencies' infrastructures are essentially run by contractors with a few GS personnel making bad decisions every chance they get and a few DoD personnel acting like their rank can fix technical problems.

Re:AWS us-east-1 outage by sneak on Tuesday December 07, 2021 @08:02PM (#29480199)
I'm not talking about running infrastructure, I'm talking about working with HR and making sure that the people they've hired as sysadmins aren't meeting with FSB or PLA agents in a local park on weekends and accepting suitcases of USD cash to accidentally `rm -rf` all of the Zookeeper/etcd nodes at once on a Monday morning.
Re:AWS us-east-1 outage by garbagecoder on Tuesday December 07, 2021 @11:49PM (#29481375)
I doubt it. We’re shit at critical infrastructure defense and the military cares mostly about its own networks. And industry really doesn’t want to cooperate. I was in cyber and military IT. Can’t speak for the IC, but I really doubt it.
Re:AWS us-east-1 outage by dylan604 on Tuesday December 07, 2021 @05:14PM (#29478907)
> I would imagine that the US IC (CIA/NSA) already does some free consulting for these giant companies to this end,

Haha, it would be funny if the IC reaches out to BigTech when failures occur to let them know they need not be worried about data loses. They can just borrow a copy of the data IC is siphoning off them. /s?

Re:AWS us-east-1 outage by taf2 on Tuesday December 07, 2021 @03:31PM (#29477814)
I wouldn't jump to say it's short sitedness (it is shitty) but it could be a matter of being pragmatic... It's easier to maintain the code if it is loaded at run time (think thin client browser style). This way your iot device can load the lastest code and even settings from the cloud... (advantage when the cloud is available)... I think of this less of short sitedness and more a reasonable trade off (with shitty side effects)
Re:AWS us-east-1 outage by outime on Tuesday December 07, 2021 @03:35PM (#29477873)
Then you could just keep a local copy available as a fallback in case the latest code cannot be fetched. Not doing the bare minimum and screwing the end user isn't acceptable IMHO. But I also understand that'd take some engineer hours and report virtually no benefits as these outages are rare (not sure how Roomba's reliability is in general on the other hand) so here we are.
Re:AWS us-east-1 outage by epistasis on Tuesday December 07, 2021 @03:39PM (#29477922)
I don't think that's ever a reasonable tradeoff! Network access goes down all the time, and should be a fundamental assumption of any software.

Maybe I'm too old, but I can't imagine a seasoned dev, much less a tech lead, omitting planning for that failure mode

Re:AWS us-east-1 outage by thayne on Tuesday December 07, 2021 @03:35PM (#29477876)
there's a reason it is called the Internet of things, and not the "local network of things". Even if the latter is probably what most customers would prefer.
Re:AWS us-east-1 outage by 9wzYQbTYsAIc on Tuesday December 07, 2021 @03:37PM (#29477889)
There’s also no reason for an internet connected app to crash on load when there is no access to the internet services.
Re:AWS us-east-1 outage by mro_name on Tuesday December 07, 2021 @04:05PM (#29478217)
indeed.

A constitutional property of a network is it's volatility. Nodes may fail. Edges may. You may not. Or you may. But then you're delivering no reliabilty but crap. Nice sunshine crap, maybe.

Re:AWS us-east-1 outage by fulafel on Wednesday December 08, 2021 @01:42AM (#29481903)
In a traditional internet style architecture, your Roomba and phone would both have globally routable addresses and could directly communicate - your local network would provide hosts with internet connectivity.
Re:AWS us-east-1 outage by garbagecoder on Tuesday December 07, 2021 @11:46PM (#29481348)
The internet was originally designed to still work if there was a nuclear strike on a node. Now we can’t even cope with us-west-1 down.
Re:AWS us-east-1 outage by codegeek on Tuesday December 07, 2021 @04:05PM (#29478214)
"If you can't connect to the cloud, fail, move on and load the app so that local things are allowed to work."

Building fallbacks require work. How much extra effort and overhead is needed to build something like this ? Sometimes the cost vs benefits says that it is ok not to do it. If AWS has an outage like this once a year, maybe we can deal with it (unless you are working with mission critical apps).

Re:AWS us-east-1 outage by dylan604 on Tuesday December 07, 2021 @05:16PM (#29478924)
Yes, it is a lot of work to test if response code is OK or not, or if a timeout limit has been reached. So much so, I pretty much wrote the test in the first sentence. Phew. 10x coder right here!
Re:AWS us-east-1 outage by mananaysiempre on Tuesday December 07, 2021 @06:10PM (#29479351)
Remember the Signal outage? Which is to say, the testing is the problem, not the implementation itself. (Which isn’t to say I think it’s optional—I myself generally don’t run online-only local-by-their-nature things.)