News for hackers. Stuff that matters.
Re:Major AWS Outage Happening by radium3d on Monday October 20, 2025 @03:09PM (#45647887)
Once you've had an outage on AWS, Cloudflare, Google Cloud, Akismet. What are you going to do? Host in house? None of them seem to be immune from some outage at some point. Get your refund and carry on. It's less work for the same outcome.
Re:Major AWS Outage Happening by CobrastanJorji on Monday October 20, 2025 @04:11PM (#45648679)
Multi-cloud. It's fairly unlikely that AWS and Google Cloud are going to fail at the same time.
Re:Major AWS Outage Happening by radium3d on Monday October 20, 2025 @04:19PM (#45648791)
Yeah, just double++ the cost to have a clone of all your systems. Worth it if you need to guarantee uptime. Although, it also doubles your exposure to potential data breaches as well.
Re:Major AWS Outage Happening by jancsika on Monday October 20, 2025 @05:02PM (#45649354)
> double++

I'd suggest to ++double the cost. Compare:

++double: spoken as "triple" -> team says that double++ was a joke, we can obviously only double the cost -> embarrassingly you quickly agree -> team laughs -> team approves doubling -> you double the cost -> team goes out for beers -> everyone is happy

double++: spoken as "double" -> team quickly agrees and signs off -> you consequently triple the cost per c precedence rules -> manager goes ballistic -> you blithely recount the history of c precedence in a long monotone style -> job returns EINVAL -> beers = 0

Re:Major AWS Outage Happening by radium3d on Monday October 20, 2025 @06:45PM (#45650382)
Lol :)
Re:Major AWS Outage Happening by dexterdog on Monday October 20, 2025 @06:25PM (#45650213)
And likely far more than double the cost since you have to use the criminally-priced outbound bandwidth to keep everything in sync.
Re:Major AWS Outage Happening by unethical_ban on Monday October 20, 2025 @05:48PM (#45649850)
Shouldn't be double in the long term. Think of the second cloud as a cold standby. Depends on the system. Periodic replication of data layer (object storage/database) and CICD configured to be able to build services and VMs on multiple clouds. Have automatic tests weekly/monthly that represent end-to-end functionality, have scaled tests semi-annually.

This is all very, very hand-wavey. And if one says "golly gee, all our config is too cloud specific to do multi-cloud" then you've figured out why cloud blows and that there is no inherent reason not to have API standards for certain mature cloud services like serverless functions, VMs and networks.

Edit to add- ink ow how grossly simplified this is, and that most places have massively complex systems.

Re:Major AWS Outage Happening by jimbob45 on Tuesday October 21, 2025 @12:04AM (#45652374)
And data egress fees just to get the clone set up, right? This doesn’t seem feasible as a macrostrategy. Maybe for a small number of critical services.
Re:Major AWS Outage Happening by yeswecatan on Tuesday October 21, 2025 @09:51AM (#45655793)
How do you handle replication lag for databases?
Re:Major AWS Outage Happening by zacmps on Tuesday October 21, 2025 @11:08AM (#45656772)
If you use something like cockroachdb you can have a multi-master cluster and use regional-by-row tables to locate data close to users. It'll fail over fine to other regions if needed.
Re:Major AWS Outage Happening by cakeday on Monday October 20, 2025 @05:43PM (#45649796)
> Host in house?

Yes, mostly.

Re:Major AWS Outage Happening by grogers on Monday October 20, 2025 @06:24PM (#45650196)
Certainly if you aren't even multi-region, then multi-cloud is a pipe dream
Re:Major AWS Outage Happening by bean469 on Tuesday October 21, 2025 @01:57AM (#45652901)
> What are you going to do? Host in house?

Yep. Although it's just anecdata, it's what we do where I work - haven't had a slightest issue in years.

Re:Major AWS Outage Happening by nxpnsv on Tuesday October 21, 2025 @02:22AM (#45653010)
Cheaper, faster, in house people understands what’s going on. It should be a given for many services but somehow it’s not.
Re:Major AWS Outage Happening by Breza on Tuesday October 21, 2025 @02:31PM (#45659700)
I totally agree with you. Where I work, we self-host almost everything. Exceptions are we use a CDN for one area where we want lower latency, and we use BigQuery when we need to parse a few billion datapoints into something usable.

It's amazing how few problems we have. Honestly, I don't think we have to worry about configuration issues as often as people who rely on the cloud.

Re:Major AWS Outage Happening by erikpukinskis on Tuesday October 21, 2025 @09:05AM (#45655268)
On premise? Or do you build servers in a data center? Or do you lease dedicated servers?
Re:Major AWS Outage Happening by Breza on Tuesday October 21, 2025 @02:28PM (#45659659)
Not GP, but my company also self-hosts. We rent rackspace in a colo. We used to keep my team's research server in the back closet before we went full-remote.
Re:Major AWS Outage Happening by bean469 on Tuesday October 28, 2025 @01:04PM (#45735570)
We have our own data center with servers. The upfront costs are high, but it was worth it in our use-case
Re:Major AWS Outage Happening by Breza on Tuesday October 21, 2025 @02:34PM (#45659729)
Why not host in house? If you have an application with stable resource needs, it can often be the cheaper and more stable option. At a certain scale, you can buy the servers, hire a sysadmin, and still spend less money than relying on AWS.

If you have an app that experiences 1000x demand spikes at unpredictable times then sure, go with the cloud. But there are a lot of companies that would be better off if they seriously considered their options before choosing the cloud for everything.