Speaking from someone who moved a SaaS operation from co-lo to AWS. Everyone (especially the C-suite) was much happier.
Everything from buying the HDD, to his hours to the parking ticket at the data-center. He convinced me and the point was clear: a single failing HDD would costs us more than renting similar computing power for over 6 years. We had two such failing HDDs in three years prior.
This "1 server" is equivalent to at least two part-time admins to ensure smooth operations. No biggie if said server just hosts a static website, big deal if runs mission critical processes.
That's the part too many "self-hosting" advocates conveniently leave out. A correct TCO calculation must include this cost (and proper risk assessment) and only then can a comparison be made.
As a business you need to ask yourself: and who pays for the time it takes to educate at least two employees on this "basic and elementary matter"? If it really were so "basic and elementary", how come even critical infrastructure regularly falls victim to the most primitive of ransomware attacks?
Misconfigured email and network access settings, vulnerable, out of date software stacks, non-existing contingency plans, lack of monitoring, no escalation strategy, etc. etc. Billions are lost each year due to this.
It's very easy to underestimate the actual complexity of the task in its fullest whilst overestimating one's own abilities because "it's not hard to setup NGinx and service stack XY on a random x64 box".
Anecdote from personal experience: tasked an applicant for a DevOps position with comparing two systems we could choose from for a backend service. The systems came in the form of installation packages and docker-images respectively.
Having had no experience with Docker at the time, I thought it'd safe my team a great deal of time to have the "new guy", who knew Docker inside out, deal with it.
A week later we had two results: 1) a usable comparison of the two systems and 2) an e-mail from the cloud service who informed us that the machines we rented were performing DDoS attacks on a foreign company...
New guy had forgotten to properly secure the test machines he had set up and it took hackers all of 72 hours to find and hijack them. Luckily it only took a few clicks to wipe the offending boxes, but it served as a warning for what happens if you take the whole "big scary nothings" too lightly.
I accept your points, but far too many inexperienced devOps are given keys that should not have them. From your Docker story, it sounds like poor decision making on your behalf. Just because a developer knows Docker does not mean they know how to expose that securely to the public - far too many devs believe that's not their job.
While that's certainly true, I think you missed the part where the guy specifically applied for a DevOps position.
The "Ops"-part wasn't optional and exposing a test system to the public in the first place is a big fail in and of itself obviously.
But that's besides the point: the point was that operating even a single server is more complex than many people tend to admit and you just confirmed this statement.
I also don't understand how you jump from "it's not hard" and "big scary nothing" to "funded startups" and getting a "different team" preferably consisting of knowledgeable "grey beards". Which is it? Is it simple enough to learn on a rainy weekend or do you actually need dedicated trained and/or experienced professionals, because far too many devs are unsuitable for this?
I have trouble following your train of thought tbh.
It also doesn't account for all of the out-of-hours stuff we need to do. Even with a hosted solution and a lot of paid-for services like Cloudflare, we still have to check out every unusual error email, everything that seems out of the ordinary.
If you are 200+ people, great, you probably already have a team to manage this. If you are small like 90% of companies then you won't and I want to spend my time on the things I am expert in.
Really? There is a giant population of senior experienced engineers looking to do basic a lot of basic sys admin stuff? And they're willing to work for peanuts, right?
By the way, I totally agree with your basic point. The developer probably downloaded some images from a random public repo. As you say, far too many devs think security is someone else's problem and then get upset when the security person blocks their app going into production.
If it were so basic, the DevOps person they hired should've been able to do it after taking a 6-7 hour docker course.
No need to ensure the software is up-to-date, no worries about hardware outages, no need to monitor the system, etc. etc.
The person working with a rented service infrastructure can focus on what makes them money: the process.
With self-owned hardware you need contingency plans, setup, maintenance, redundant ISPs, UPS, extra electrical setup (many offices automatically turn off power to all outlets after office hours to safe electricity and reduce the risk of fires), and much more depending on the required tasks and their importance.
If you want to know what happens if you cheap out on that because "it just works" anyway once it's been set up and operated by an underqualified and overworked person, look no further than the recent ransomware incidents.
Some of them could've been easily avoided had the IT infrastructure been up to snuff. Way too many businesses and public entities still treat IT as a necessary evil that can be handled casually, not realising their dependence on it.
The specifics differ on a case-to-case basis, but in general it's a terrible idea for a small non-IT business to operate their own IT infrastructure. The solution doesn't have to be "cloud" or SaaS either. Hiring a competent 3rd party to deal with the specific implementation and operation for you is just as good even said party is "Bob's Business IT Services" and not AWS or Microsoft Azure.
> No need to ensure the software is up-to-date, no worries about hardware outages, no need to monitor the system, etc. etc.
> The person working with a rented service infrastructure can focus on what makes them money: the process.
No, that's just not even remotely true. VMs, managed services all require updates that are pushed to the end user of the service. Downtimes happen, and dealing with those are pushed to the end user of the service by cloud providers far too often. I have countless support tickets at all levels of the cloud stack of back-and-forth on issues, many as simple as "how about serving 200s, not 500s?"
I wish managed services were a rainbows and unicorns world of "I give you cash, you give me magic computer thing that scales and never goes down" … but that just isn't the case today for many cloud services. (I think about the one service I'd be willing to pin such a medal on is S3. It comes remarkably close, but even there, I've had an incident where we had an outage due to S3 that took much back & forth with support to finally get an engineer on their side to look at it.)
When I say "No need to ensure the software is up-to-date, no worries about hardware outages, no need to monitor the system, etc. etc." I don't mean these things don't need to happen at some point. The idea is that it's not you, as in the customer, who's responsible for this.
If I book a Redis-DB of a given size, I don't have to care about its software version, proxies, what OS it's running on, how big the VM is, what the specs are, and so forth. That's what I'm getting at.
It's far higher level than individual VMs and it doesn't mean downtimes don't happen either. It simply means it's not you, the customer, who has to troubleshoot and fix them.
And what I'm getting at is that you do. While there might exist managed services that don't require this, many managed services nonetheless do. E.g., AWS RDS regularly has updated versions, and it is somewhat up to the customer to schedule them & apply them, and downtime happens during that. RDS is relatively pain-free, and is higher on my list of "would use again" for that reason. But take something like Azure AKS: it's a "managed" service, but upgrades are very much not pain free. I have to manually schedule them, and manually apply them, and deal with fallout (which is often, IME).
> It simply means it's not you, the customer, who has to troubleshoot and fix them.
Again, I disagree. When S3 had its outage, I had to debug why certain requests weren't happening, I had to capture sufficient request IDs to prove to AWS that S3 was in an outage for us, and only then did we get the proper support, and eventually, attention from engineering. My company had to pay an engineer to do that. Sure, I'm not literally debugging S3 actual, but that doesn't mean an engineer isn't needed on the consuming side. Same story with an example with ACR where we were receiving 500s: it took not only capturing 5xx requests & their time of occurrence, it took multiple back and forths to convince Azure that the 5xxs were on their side. (Both of these incidents also demonstrated a real fundamental lack of alerting on the side of the companies managing these services. Our monitors on their services very clearly indicated outage. S3's excuse was that it was "just" our bucket. But the point is that they have no insight into the actual service the end user is perceiving even after the customer raises the support ticket.)
Just recently I was SSHed onto an AKS VM to try to help Azure understand why packets weren't making it from their Virtual network into AKS. Data, from one Azure service to another! (And we didn't figure it out, either; we ended up changing what we were doing anyways, and so the investigation became moot.) And this demonstrates another issue I've hit, a bunch, is that interactions between managed services are also impossible to debug: you end up with two separate services both claiming it isn't them that is out of spec, but nonetheless the end result doesn't work, and nobody on either side has the competence to actually debug the problem.
If you want the downtime to end, it becomes you, the customer that has to debug & troubleshoot these things. And that is a cost that must be accounted for when weighing managed services vs. unmanaged ones.
However as all developers are AWS experts (at least according to their resume), you have to pay that price regardless
It is always the 1% of difficult stuff that hits you, but it is very rarely the correct call to optimize for that 1% instead of for the 99% of uses.
But the number sounds a bit off… I have fuzzy memories of some fun around that many concurrent requests with Apache 1.3 — that is, one process per user. Doesn't the fun begin quite a bit higher with just any more modern software?
You tell me, I'm not a sysadmin:) I can only attest to the fact that without any kind of RAID or suitable hardware setup (RAM for caching, fast networking etc.), even a dozen people accessing the same file can be very problematic (provided the file is large enough).
As for the use case: digital content creation comes to mind. Sharing assets globally especially in times like these when more people are working remotely is such use case.
Now it isn't cheap, but the five year ownership cost difference compared to iron doesn't cover a month of my salary, so.
Do the math and see how using AWS just doesn't make sense at the scale most companies operate.