For any individual company able to offload the blame, that's great. It's not so great if half the countries' doorbells, robot cleaners, various home streaming service setups, the baby camera, the fridge, the smart TV and your phone stop working... All at the same time.
In my opinion, the 'downtime' really should be measured in $NUM_SERVICES_STOPPED X $TIME, instead of just $TIME. And in this case I think any long time Amazon outage is orders of magnitudes worse than your regular old slow IT company outage.
Similarly, I've had EC2 instances run for years without ever being rebooted or going down. One of them's still running today after 5 years.
But none of those services were being used 24/7; if the internet went down for an entire weekend, or a hard drive was a little bit corrupt but kept running programs in memory, I probably would never have noticed. I've also had EC2 instances literally just fall off the map and sort of disappear, and had them manually replaced without notice by AWS, had virtual drives fail and corrupt, and had calls to services fail. And I've had my desktop's power supply get fried by a power surge.
Without a lot of experience, running systems seemed easy. But as time went on I learned that it can be easy, and it also can go down if somebody blows on it the wrong way. What we see as being reliable may just be chance. The only way to guarantee reliability is to expect that things are going to go down, and design and build it accordingly.
What AWS makes easy is they give you all the components for reliability, but you have to do the plumbing yourself. I'll bet you the people whose services went down did not properly design for reliability, as they were probably running in one region, in one set of AZs, and relied on distributed system operations that can fail, and didn't properly account for how to deal with those failures. One product I maintain on AWS did not go down, but another did.
Also, the more components a system has, the higher probability there is of failure. Big systems are actually more error prone than small ones.
1. How easy can I access your physical servers ? 2. What happens if there is a catastrophic failure, for example local power outage or a major flooding 3. How secure is your server? Are you regularly patching your operation systems 4. If I want to run a project that requires double the capacity of your current hardware for a specific project, how long is it going to take to get it spun up?
- Regularly patching is automated and took about 30 seconds to configure, using an automated script
Regarding running your own physical servers, that is a different ballgame, but for all of my projects, if I need to:
- I can pretty easily spin up VPSs / bare metal servers anywhere (netcup, linode, hetzner, etc) and provision there while I wait for new hardware to come in - If you want to double the capacity of your current hardware, you'll have to order it and wait, but it's cheap (vs the major cloud providers) to way over provision if you're running your own physical hardware, so you can pretty easily have 2-4x extra capacity and still come out with extra money in your pocket.
I host in the cloud, but I think people vastly over estimate how much it saves 90% of cloud customers.
There are well-understood answers to all your questions, they are not too difficult, they just cost money - some businesses choose not to spend that money, some weigh cost-benefit and go for AWS, some decide to go for in-house servers, some go for hosted Virtual Servers, some go for serverless.
Why does everything have to be built one way?
I'd also say some choose not to spend the money, but fail to consider the cost of that choice.
For example: Doing old-school manual deployments that require herculean efforts to update at off-hours on the weekends burns people out, and makes it hard to attract new talent. In other words, you've made the decision to spend more money on finding and retaining people. But it's definitely way cheaper to pay for colocating a single Dell server you bought 3 years ago than what you'd spend in the same time on AWS.
And if your hardware never dies, paying for the redundancy might seem silly. A lot like paying for fire insurance despite the fact your house has never even burned down.
Of the options I mentioned, you seem to have picked only self-purchased hardware with no redundancy and no backups (your addition) to compare with, and also throw in manual deployment (why?).
Most businesses are somewhere between a forgotten old dell server in the closet and fully hosted multi-region auto-scaling fully bought in to AWS, and that's OK!
1: Access to the data center requires either an access card plus biometric verification (fingerprint in one DC in my case, retina scan in the other), or an ID-verified appointment. Then, you still need to know where my server is, plus the access code for the rack (or, you need to be an average lockpicker, but beware, there's cameras...).
2: Each data center has dual-provider AC feeds, plus generators, and provides A/B feeds to my rack. I've not had a dual-feed outage in the last 20 years or so.
3: No cloud provider that I'm aware of guarantees server security or does automated patching (for servers, not services). So, keeping your server up-to-date seems equally important for both cloud and non-cloud scenarios?
4: At least two weeks, I guess? I have sufficient VM host capacity to accommodate 30% unplanned growth, but 100% would require new hardware. So: ordering two servers, installing these in two data centers. But if the new project also requires significant bandwidth, getting new Internet connections in might take longer.
Look, I'm definitely not denying that "the cloud" makes it easier to scale fast, but scaling fast is not an overriding concern for most businesses. Cost is, and self-hosting, even with a pretty redundant infrastructure, is still much cheaper than AWS.
Getting higher uptime is super easy for smallish inhouse deployments.
You just don't install any updates and let the server run, trusting your VPN to shield you from possible security issues.
The maintenance burden is the reason why people often prefer the cloud services, not the uptime. Because maintaining the instance with updates, reading all patch notes and steps for migration, keep every health metric monitored and respond quickly on issues without getting stuck googling for possible reasons is quite a bit of work and quickly forces you to employ n+1 people.
We run some bare metal servers. They just never go down. Solid continuous pings for years as monitored from elsewhere on the Internet. That's because they're just boxes on a rack somewhere running an OS and some steady-state services (ZeroTier roots). Simplicity is more robust than complexity.
SaaS is definitely about the pain of managing and upgrading software, but it's also about OPEX vs CAPEX. Many companies will pay more for things to put them in the OPEX column for various entirely synthetic accounting, investor relations, and tax reasons.
I do wonder if the pendulum there will swing back though since if you price out cloud vs. physical hardware the market has become extremely distorted. Many companies spend enough on AWS to buy an entire rack of hardware at a different data center every month and pay 2-3 employees to manage it. That hardware would be up to 100X as fast and powerful as what they rent at AWS and bandwidth would be almost free. That's a really distorted market. The amortized costs should not be this different.
I think the issue here is that it's not zero-cost to switch. Your processes will adapt to some implicit assumptions that aren't true outside AWS, Azure, or whatever vendor you locked yourself into.
If we somehow managed to have a completely standardized interface here, the market would be more competitive.
They can try to saturate the VPN host maybe, but that's going to be challenging considering that it's going to be limited to connection requests without valid credentials.
and these are likely set to be ignored on multiple failed attempts through fail2ban or similar tooling
A long time ago, I used to circulate snarky little emails at work.
One of them was responding to this very concept.
My managers were throwing out our working and mature UNIX servers (implementing DNS, Mail, and other services) in favour of NT.
The new system crashed a lot, we had some security breaches, but at least it was 'industry standard.'
Managements' response was that with the UNIX stuff we had no one to pin our outages on. Now we could blame Microsoft, and call their support line.
I circulated an email making fun of this justification, which promoted a fictional product called 'Blame Studio' which would help you map out the blame path for any of your products or services.
It would help to make sure that none of the blame ever landed on you, but rather was always redirected onto some other company.
"my outage means the customer is probably also down so they maybe don't care" is not a viable way to run a business in my mind.
Yes they definitely are.
Let no more have your integrations with systems in the cloud frustrate your customers when they are unreachable - let the news explain the downtime. JOIN the Downtime Umbrella NOW and receive 5 downtime lever pulls for FREE! /s
Your customers will be more understanding if your outage is part if a wider outage that makes national news.
Any services you integrate with are likely down too. If two services with 99% uncorrelated uptime together drops to about 98%. It doesn't drop if the downtime is perfectly correlated.
Even if you don't directly integrate, your customer's workflow might. They see many services down and say, "cool, time to get caught up on laundry". If only you're down it's more aggrevating.