But [to state the obvious but sometimes overlooked] you don't just point an AWS account at the company git repo and walk away.
There's a lot of work and expertise needed to keep AWS setup up and running, so you already have to hire people.
At a modest size startup we already have close to ten people DevOps team to manage AWS. That same size team could easily keep bare metal servers running. At our scale it's still a bit cheaper to be on AWS, but not too far in the growth curve it'll start to become cheaper to be on bare metal.
Admittedly this was hopefully only going to apply for a few years until they finished rearchitecting everything, but I doubt it would ever reduce back down to anywhere near the original costs.
The equation is also mirrored for something that started and grew on AWS though - going to bare metal means building up tooling and processes you didn't have before. The transition will be expensive in either direction.
I thought this was pretty weird since the original value proposition (AFAIR) was to reduce costs/head count. But everywhere I’ve worked that used AWS, had specialists employed to manage AWS. And I think the value proposition is more that it’s easier to find people who know AWS - because there are training providers and certifications - than people who know how to do everything themselves.
Once you get to a certain size I think you can attract a team who can build out rather than buy in, and in so doing reduce costs.
[1] https://www2.deloitte.com/au/en/pages/economics/articles/eco...
I don't think this true for the vast majority of applications. It's probably true for the Youtube, Netflix, OneDrive, etc critical data path, but when your ownership is more like two dozen applications ranging from tiny (one consumer on some infrequent interval) to mid sized (under 100k requests a minute, for text, json, images, generated UI components), it's damn near impossible to beat from a pricing standpoint.
Going back to on prem servers, even doing kubernetes/docker orchestration would likely increase my current team's monthly capital expenditure from ~$14000 a month to somewhere in the realm of 50-60k a month. The sad thing about those prices are that our on prem offerings are very cheap compared to the last price I worked, a $BIGBANK that has multiple data centers in multiple countries. As an internal client with the same footprint we have now, $BIGBANK global infrastructure folks provisioning would take $90-110k and 6 to God even knows how many months depending on how fast you could get through the churn of governance, compliance, security, and controls.
Edit: just realized I forgot to mention I'm talking about serverless projects almost exclusively. That's important to the cost savings, because I agree that I'd rather host something in the 9 year old server in my basement and run my own reverse proxy than use EC2/EKS from a cost perspective.
It is absolutely easier to use S3 than to create your own fast, highly available, infinitely scaling storage solution on your own metal. It requires more than zero knowledge / expertise to use S3, but far less than it would to implement and run yourself.
If you can accept that, then we already agree in principle. It's just matter of where the line is drawn for various services and use cases.
It’s hard to quantify how best you’ll be served but a lot of people are following the mantra of “nobody got fired for going AWS”.
It makes sense for some people, others are cargo-culting; yet more are fanning the flames of that cargo cult because their pay check depends on it.
Sysadmins are/were paid much less than cloud native devops people, and you need the same number of them unless you keep things very simple, which cloud providers do not incentivise. One need only look at AWS reference architectures.
You are certainly correct about the the overly complex AWS reference architectures. I've seen relatively simple applications with just as much infrastructure code (generally "terraform", occasionally CloudFormation JSON) as application code. It's crazy.
Most automation that you know of as “devops tools” are borne from sysadmins.
Terraform was written by a sysadmin; puppet, ansible, saltstack and cfengine too.
It is revisionist to think sysadmins were not automating their jobs.
Old school sysadmins used to know C, bash and Perl.
New school devops just traded that for a handful of DSLs and Python.
I’m sure some technicians working the IT department managed to get by running scripts created by sysadmins and claiming not to code, but it was definitely the common case that 20 years ago sysadmins could code and worked tirelessly to “automate themselves out of a job” (literally a mantra i was told as a sysadmin 15 years ago)
The people who wrote tools like terraform and ansible were engineers with system administration skills. Those are very rare.
The pay increased, the titles and tools changed but the mentality didn’t.
People just started beating their chest about devops and pooping on the legacy of sysadmins which is what most devops/SRE are.
Sounds like they were automating though.
“Cattle not pets” was a sysadmin mantra, but the business wanted pets most of the time.
I’m with you on most of your post, but this is straight up wrong.
https://www.linkedin.com/in/mitchellh/
Or are you saying that he's not the author? Commits seem to suggest otherwise.
commit fb9c58f0e20a85a5358142ddb39a2f74d3bbe69b
Author: Mitchell Hashimoto <[email protected]>
Date: Fri May 23 11:03:38 2014 -0700
config: better error message
commit 089822a36f0a5e4c583ab401b7496b48bfa31d65
Author: Mitchell Hashimoto <[email protected]>
Date: Fri May 23 10:52:19 2014 -0700
config: some comments
commit ec3f72703c82f50787ef86498d856a5798900ad8
Author: Mitchell Hashimoto <[email protected]>
Date: Thu May 22 16:56:28 2014 -0700
Initial work on config
commit 649cf336e86da10910f6e50ea3cfaad507a2e336
Author: Mitchell Hashimoto <[email protected]>
Date: Wed May 21 16:28:53 2014 -0700
Initial commitAn operations engineer and a sysadmin are two entirely different roles.
A system administrator administers systems while an operations engineer dedicated their time writing code to automate processes.
Sysadmins automate their work. Operations Engineer is just another title for sysadmin.
Systems Engineer is also another title for the same thing.
Platform Engineering (depending on where you are) is yet another variation.
If you're responsible for the operating system, monitoring, clustering, hardware etc; then you're firmly in "sysadmin" land.
Pedantry is requisite.
So do janitors. What's your point?
> Operations Engineer is just another title for sysadmin.
It really isn't. That makes as much sense as claiming that DevOps is just a sysadmin that's also a developer.
Many "operations engineers" are actually sysadmins. Many "devops" are really sysadmins also doing development.
That's not true of any place I experienced in the early to late 90s. If you meant earlier, perhaps, I wasn't there.
The growth of perl, for example, was in great part from the sysadmin community automating everything.
Remember that in those days you likely only had one machine. By "you" I mean the whole department. Everyone was logged into it and it handled email, talk, documents, compilation and debugging, etc.
That doesn't take away from the fact that the sysadmins were automating everything they needed to do, in the enviornment that existed.
The idea that I would need to automate the installation of an OS and applications on a fleet of machines was never contemplated because it didn’t make sense. I had one machine and OS upgrades arrived in the post every 6 months or so - on cartridge tape.
It was about need, not competence.
I worked in the mid-00s and it wasn’t a lot better in the dev space. People passing USB keys to each other was pretty common, SVN and CVS were around but there was a lot of developer code outside of it and peer review was going to someone’s desk and walking through the new code, nothing at all like what we have today for processing change requests.
You’re talking about systems administration when it was going from pets to cattle, the primordial period where people were automating but not applying software development practices on themselves yet because even the software development practices weren’t well defined.
Sysadmin was always lagging 5y behind development w.r.t. programming practises. The same is true today of devops.
tell me how many terraform repositories have unit tests or infra scripts for that matter which would be much simpler.
I did know some fantastic sysadmins at a local university around that time period. They had huge labs of Alphas and Suns and automated everything! Unfortunately, those folks didn't go into industry.
The closest I’ve seen to devops nirvana has been NixOS, but its frustratingly divergent terminology and file formats make it very hard to cross the chasm.
But nowadays if you want to manage your own metal then you can use those tools as well. You can still run hypervisors VMs on your own servers. You're just not paying Amazon 6x the price of what it would cost you, and you're not relegated to low-performing compute, memory, and IO elements that Amazon standardized on.
I’ve worked at exactly one storage company that had a customer that had exascale data. They were doing cancer research as I recall and their test machines generated a lot of data. I heard stories about CERN at conferences but they also self host their data.
But those were outliers. All of the large and small enterprises outside of that could fit all of their “big data” in the memory of a single blade server and still have plenty to spare. You can get machines these days with many TB’s of RAM.
I can tell you that massive amounts of storage are used by eg Mercedes Benz during simulations they need when developing cars. Back when I was reading about this - maybe a decade ago - they were using mind boggling amounts of storage. Still all before SSDs. You can assume all other world-class engineering firms in every kind of difficult engineering that requires physical robustness or safety have a similar need. But Mercedes Benz at least used an on-premises solution, simply because storing stuff on AWS would have created latency and throughput issues.
Is the issue big data or small machine?
Memory size of the largest AWS/GCP/... instances is a good indicator for small machine.
It doesn't mean I would go for the large machine. It just means I won't subscribe to "big data" as a reason to do X.
And it forces me to recalibrate this boundary regularly. Looks like double digit TB is currently the memory boundary for renting. I was still on the single digit TB train.
MinIO by itself is fantastic software for sure, but running storage appliances is shades of difficult or expensive.
On the one hand you can buy a netapp filer and support plan which is basically as self-healing as the cloud is, netapp will send a human to go replace failed drives before you even know a fault is imminent.
On the other, that's expensive but running your own is complex if you're not setup to do it already.
MinIO is but one part of a data storage puzzle. Though a very important one.
Try uploading 5PB of content and downloading it that day at 1M RPS.
Huge difference b/w reading a few pages of documentation on secrets manager, and clicking a button to get the service on vs deploying, and maintaining a vault on bare machines.
It's much easier to setup and forget basic bare metal servers with PG/NGINX and whatnot, than it is to automate using dozens of AWS services.
People pretend that AWS doesn't cost engineers to run it, when it's IMO basically the same human cost, if not bigger as complexity grows. You just don't pay that cost upfront, but you sure do pay it later with interest.
You get stuff like HA but that's not free. You also now have to manage a new boatload of services, scripts, changing APIs, etc.
3 years fast forward with hundreds of customers, and we haven't had a need to hire someone full time to "manage" these services. This may/will change down the line, but I couldn't have built my business without one of these cloud vendors.
Edit: I also know the pain of building (and managing) this on bare metal (last company did that) - and it was just apache/php/mysql on bare metal. It was a mess
This is a reference to the country Brazil [0], which centers on...
This is a reference to "the rule of the desk" [0], and "an elaborate, confusing structure", which centers on a monster [1], in this case a bureaucrat at a desk
[0] https://www.workhuman.com/blog/a-brief-history-of-bureaucrac...
I guess you've never done it yourself?
I'm not sure what those 4 engineers would be doing. You purchase a few servers with 5 year on site warranty and remote management, you take a couple of days to install them in some racks, and you never visit the site for 5 years. If they break, the manufacturer sends someone to fix them on site. The rest of the time you administer them remotely - just like you would AWS. If you want VM's install proxmox.
Whether it's cheaper than renting bare metal in a data centre is an debatable - colo costs a small fortune where I live.
But they aren't comparing it to that. They are comparing it to locking themselves into a closed source system that need specialised expertise to run, versus administering an open source Linux system that even the people running EC2 should be very familiar with. Most of the stuff AWS provides has open source counterparts - hell a whole pile of it is just open source they wrap a proprietary API around and charge you for.
10 times sounds like a lot, I'm guessing it's really less. Even so I come form the camp that shakes his head in disbelief at what people will pay AWS for basic VM and storage services, dressed up in a fancy API and marketing.
I know *data centers* that run on a few naive 20 yos admins/technicans + 1-2 "engineers" and all of them combined do receive salary of $5-10k/month in east eu
One of the companies had a picture of one of their "datacenters." It was something like 10 racks in a moldy unfinished basement with visible water on the floor of what I'm guessing was a residential building. Maybe they had mopped the floor for the picture?
I thought it was strange they would put that picture on their site and not some GCI or something. I guess at least you know they are not lying about being a real place?
they've decent building, data center tier, so power/network redundancy, etc, etc
they're just cheap as fuck when it comes to people
I really should start a blog for some of the weird research I do sometimes.
If they saved $800k per year, and they have to hire four additional ops engineers to run it at a cost of $400k per year, then they actually saved $400k. Which is still substantial and, all else being equal, sounds worthwhile.
If they saved $800k per year, and they have to hire ten additional ops engineers to run it at a cost of $1 million per year, then they've actually gone and burned $200k on something that provides no additional value to the business or their customers.