News for hackers. Stuff that matters.
Re:AWS down again? by PedroBatista on Friday December 10, 2021 @09:10AM (#29510010)
Only the most informed and rational companies will see how and why most of this "in the cloud" thing is a bad idea.

Most will accept it as a fact of life and continue to pay for it both directly and indirectly, as long as there's cheap money going around the cloud business can't do wrong.

But it's hilarious to see people indulging in byzantine "World scale" resilient systems that depend on a single vendor.

Re:AWS down again? by eterm on Friday December 10, 2021 @09:13AM (#29510038)
On the contrary, downtime is inevitable and I think I'd rather have downtime when all my competitors do too.
Re:AWS down again? by cnity on Friday December 10, 2021 @09:18AM (#29510108)
How about having uptime while your competitors have downtime?
Re:AWS down again? by the_af on Friday December 10, 2021 @09:37AM (#29510318)
If your system integrates with any external systems or APIs, it's likely they have downtime when one of the cloud giants are down, so sometimes being up is not so relevant. If your system is up but everything you depend on is down, how useful is that?

For (almost) entirely self-contained systems it can still be useful, of course. But everything wants to be interconnected to everything these days...

Re:AWS down again? by spoils19 on Friday December 10, 2021 @02:28PM (#29514364)
That's why we proudly don't integrate with any third-parties at all, and all of our stuff is developed and hosted in-house. Even "trivial" things like email or billing.
Re:AWS down again? by scrollaway on Friday December 10, 2021 @09:24AM (#29510161)
Which, given that downtime is inevitable, leads to you having downtime while all your competitors have uptime.

Have fun with that…

Re:AWS down again? by teitoklien on Friday December 10, 2021 @09:35AM (#29510302)
If your non-cloud uptime is even 50% more then the uptime cloud offers, then you’re throughout the year, up longer then others.
Re:AWS down again? by leghifla on Friday December 10, 2021 @09:41AM (#29510373)
If you are 3 competitors on your market and you might lose x orders while your are down, but gain 2*x orders when the other 2 are down, you are ahead (with the same probability of going down as the others).

Real world is not so easy though...

Re:AWS down again? by 41209 on Friday December 10, 2021 @09:22AM (#29510143)
Would you rather have to fix your own data center, or wait 4 hours.

AWS works 99% of the time,plus it's someone else's problem

Re:AWS down again? by El_RIDO on Friday December 10, 2021 @09:58AM (#29510608)
I absolutely prefer to have the option to go into a datacenter in a hurry and actually fix stuff and be in charge, then be stuck with having to wait an indefinite amount of time, twiddle my thumbs, apologize to customers and hope for the best.

While I considered myself a decent Windows NT admin, 20 years ago, the reason I went all in on Linux and FLOSS software at the turn of the century was because I dreaded the powerlessness that these proprietary solutions gave me, when they failed. You'd call the vendors, pored through logs, finding obscure, undocumented error codes, etc. With FLOSS and self-hosting youve got all the information at your fingertips. And if you encountered bugs and you can dig into the sources, patch them, re-compile and fix things - and share them with others and feel that you're contributing to our profession.

When I got the chance to do cloud projects over the last 5 - 10 years, I always took these opportunities, hoping to ensure that I keep up with the tech. At first I was hopeful to offload the boring ops tasks, take our config management to the next level and automate even more. With every platform I got to work with, AWS, Azure and GCP so far, we kept finding bugs in their APIs, outdated or otherwise incorrect documentation and very unpredictable performance, unless you actually can run stacks at scale to average it out (as in more then just 10 - 20 instances or a larger clustered SaaS of the cloud vendor). Many times we also encountered undocumented limits that required requesting support and waiting for approval by the cloud vendor, to get even their mid-sized resources allocated. It all works very nicely on the free-tier-eligible, smallest instances and services, if as slow and high latency as is to be expected, but as soon as you actually need some decently sized storage, compute or bandwidth, it becomes quickly more expensive than what you can put in two or three datacenters for redundancy yourself, if you look at the yearly costs. So far none of the PoCs I was involved with ever got approved long term. They mostly end up as reference implementations for our customers or show case material for the corporate blog. :-(

No thanks, I don't want to go back to feel that powerless as I did on closed systems ever again. Luckily, although many seem to think that cloud is the only option to run at a global scale, you can still provide lots of valuable services on the internet using robust hardware, housed in well connected datacenters.

Re:AWS down again? by 41209 on Friday December 10, 2021 @10:54AM (#29511341)
You may be correct, for some very specific use cases. No matter how much corporate clients want to holler, a few hours of downtime isn't going to hurt anyone.

I know I'd rather someone else do it, then having to drive 2 hours away to a data center at 4:00 in the morning. But I don't know exactly what you're working on , I definitely can't imagine some use cases where a few hours of down time is just unacceptable. I know I wouldn't want to run a logistics firm with servers that go down all the time.

Re:AWS down again? by iso1631 on Friday December 10, 2021 @09:27AM (#29510188)
In my industry, five nines is the starting level. You're proposing something 1,000 times worse.
Re:AWS down again? by sumtechguy on Friday December 10, 2021 @09:51AM (#29510500)
If you have to have 5 9's starting with any cloud service is pretty tough sell at this point. You can build that type of system ontop of them. But it takes a willingness to also build your own and use competing services adding even more complexity. The BYO or competing services bit is so you can keep the lights on even when the cloud eats it, not if. Part of 5 9's is planning for catastrophic failure. Things like 'what if the load balancer/router/backhaul/dns burns out, you have a spare on hand? Then what happens to services where you had them sticky?' Even then what if both the primary and secondary are out? What then? Lots of planning and making sure you know what to do when each of those cases happen.

The nice bit though is many services can go down. Yeah it stings a bit (money, reputation, time, etc). But overall it is not that big of deal.

But for the places where you can not go down. Tons of planning and tons of backup plans with backup plans, and a different style of producing code.

Re:AWS down again? by Edman274 on Friday December 10, 2021 @11:51AM (#29512065)
Besides the fact that they used "99 percent" in a colloquial, informal way ... is that really how calculations like that work, or should work? I mean, if you have 99 percent availability, and 99.999 percent availability, maybe it's valid to say that the former is 1000 times worse than the other, but that could also be true by saying that 99.999 percent availability is 10,000 times worse than 99.9999999 percent availability (where a service only goes down for a third of a second in the span of a decade), and that 99.9999999 percent availability is infinity times worse than something that hasn't had an outage within whatever monitored period (since 100 percent availability is the same as ninety nine dot nine repeating availability)
Re:AWS down again? by stjohnswarts on Friday December 10, 2021 @12:05PM (#29512275)
Then AWS isn't really targeting you as a customer though. Most of the web really isn't that critical and can survive a few hours outage occasionally.
Re:AWS down again? by greedo on Friday December 10, 2021 @03:52PM (#29515318)
But for those who bought into AWS as a way of improving their performance and stability, "a few hours outage occasionally" can be very expensive. If my company experienced a multi-hour outage "occasionally,", it would be $100K/hour in lost revenue. Your remark reminds me of people who say "Well, Tesla really doesn't mean `autopilot` in the way you think."
Re:AWS down again? by emptybottle on Friday December 10, 2021 @09:38AM (#29510342)
You don’t need to build your own data center to host outside AWS.

You can fully manage your hardware and software fleet while still paying a host to provide data center service, and networking too.

Re:AWS down again? by jeremyjh on Friday December 10, 2021 @09:17AM (#29510085)
Do you think people who manage their own data-centers never have outages?
Re:AWS down again? by emptybottle on Friday December 10, 2021 @09:42AM (#29510396)
Running your own data center is on the far other end of the spectrum. There are numerous options between solely using AWS and building a data center.
Re:AWS down again? by jeremyjh on Friday December 10, 2021 @11:42AM (#29511931)
And people have outages in all of them.
Re:AWS down again? by marginalia_nu on Friday December 10, 2021 @09:19AM (#29510120)
When they do, they at least have the ability to do something about it.
Re:AWS down again? by mac-chaffee on Friday December 10, 2021 @09:49AM (#29510478)
My last outage was a power outage on a sunny day that lasted slightly longer than our battery backups. Without economies of scale, installing/maintaining extra generators for a small self-managed data center is prohibitively expensive.

There's always a service provider somewhere in the chain who can drop the ball.

Re:AWS down again? by whatever10 on Friday December 10, 2021 @09:30AM (#29510243)
Like shout at your IT and vent?
Re:AWS down again? by smarx007 on Friday December 10, 2021 @09:34AM (#29510281)
Which may not be good for your uptime: https://youtu.be/tDacjrSCeq4
Re:AWS down again? by tiahura on Friday December 10, 2021 @09:25AM (#29510174)
Some us remember playing Chinese fire drill and it wasn’t fun.
Re:AWS down again? by jasonlotito on Friday December 10, 2021 @09:29AM (#29510219)
No, not necessarily.
Re:AWS down again? by tyingq on Friday December 10, 2021 @09:35AM (#29510296)
>But it's hilarious to see people indulging in byzantine "World scale" resilient systems that depend on a single vendor.

As far as I can tell, there really hasn't been an AWS outage where you couldn't have avoided issues with multi-region and some careful selection of which products you're using. Which isn't much different from what you would have to do with your own infrastructure.

Re:AWS down again? by papito on Friday December 10, 2021 @09:19AM (#29510118)
Try calling "your ops guy" during Thanksgiving or when he is sloshed in a bar when your colocation servers stop responding. No one wants this job anymore. Current technology jobs have already enough unnecessary complexity to deal with uptime as well.
Re:AWS down again? by emptybottle on Friday December 10, 2021 @01:10PM (#29513315)
On the other hand try calling "AWS" in the same scenario. My money is on sloshed guy coming back with the faster response.
Re:AWS down again? by Jenk on Friday December 10, 2021 @09:27AM (#29510195)
Have you heard about the "No True Scotsman" logical fallacy?
Re:AWS down again? by stunt on Friday December 10, 2021 @11:18AM (#29511648)
I don't use Cloud for their uptime, it's all about their managed services for me.
Re:AWS down again? by Flankk on Friday December 10, 2021 @09:40AM (#29510360)
Are you arguing that purchasing and maintaining your own servers is cheaper than using AWS? Let's hear the rationale, then.
Re:AWS down again? by mac-chaffee on Friday December 10, 2021 @10:04AM (#29510678)
Yes, on-prem is generally cheaper in the long-run.

For example, my company recently purchased $400k worth of hardware that would cost $80k/month if it was all EC2. There's amortized costs of the supporting infra (cooling, power) and ongoing maintenance/depreciation, but paying for the hardware in 5 months can't be beat.

Re:AWS down again? by Flankk on Friday December 10, 2021 @11:30AM (#29511792)
You forgot the salaries of the network technicians you wouldn't have otherwise. You now have to manage attacks on your infrastructure yourself. You left out bandwidth costs as well. Your servers will need to be replaced entirely every 5 years, and partially as hardware fails.

It sounds like you're at sufficient scale that it makes sense. For most people, it doesn't make sense.