If they saved $800k per year, and they have to hire four additional ops engineers to run it at a cost of $400k per year, then they actually saved $400k. Which is still substantial and, all else being equal, sounds worthwhile.
If they saved $800k per year, and they have to hire ten additional ops engineers to run it at a cost of $1 million per year, then they've actually gone and burned $200k on something that provides no additional value to the business or their customers.
But [to state the obvious but sometimes overlooked] you don't just point an AWS account at the company git repo and walk away.
There's a lot of work and expertise needed to keep AWS setup up and running, so you already have to hire people.
At a modest size startup we already have close to ten people DevOps team to manage AWS. That same size team could easily keep bare metal servers running. At our scale it's still a bit cheaper to be on AWS, but not too far in the growth curve it'll start to become cheaper to be on bare metal.
Admittedly this was hopefully only going to apply for a few years until they finished rearchitecting everything, but I doubt it would ever reduce back down to anywhere near the original costs.
The equation is also mirrored for something that started and grew on AWS though - going to bare metal means building up tooling and processes you didn't have before. The transition will be expensive in either direction.
I thought this was pretty weird since the original value proposition (AFAIR) was to reduce costs/head count. But everywhere I’ve worked that used AWS, had specialists employed to manage AWS. And I think the value proposition is more that it’s easier to find people who know AWS - because there are training providers and certifications - than people who know how to do everything themselves.
Once you get to a certain size I think you can attract a team who can build out rather than buy in, and in so doing reduce costs.
[1] https://www2.deloitte.com/au/en/pages/economics/articles/eco...
I don't think this true for the vast majority of applications. It's probably true for the Youtube, Netflix, OneDrive, etc critical data path, but when your ownership is more like two dozen applications ranging from tiny (one consumer on some infrequent interval) to mid sized (under 100k requests a minute, for text, json, images, generated UI components), it's damn near impossible to beat from a pricing standpoint.
Going back to on prem servers, even doing kubernetes/docker orchestration would likely increase my current team's monthly capital expenditure from ~$14000 a month to somewhere in the realm of 50-60k a month. The sad thing about those prices are that our on prem offerings are very cheap compared to the last price I worked, a $BIGBANK that has multiple data centers in multiple countries. As an internal client with the same footprint we have now, $BIGBANK global infrastructure folks provisioning would take $90-110k and 6 to God even knows how many months depending on how fast you could get through the churn of governance, compliance, security, and controls.
Edit: just realized I forgot to mention I'm talking about serverless projects almost exclusively. That's important to the cost savings, because I agree that I'd rather host something in the 9 year old server in my basement and run my own reverse proxy than use EC2/EKS from a cost perspective.
It is absolutely easier to use S3 than to create your own fast, highly available, infinitely scaling storage solution on your own metal. It requires more than zero knowledge / expertise to use S3, but far less than it would to implement and run yourself.
If you can accept that, then we already agree in principle. It's just matter of where the line is drawn for various services and use cases.
It’s hard to quantify how best you’ll be served but a lot of people are following the mantra of “nobody got fired for going AWS”.
It makes sense for some people, others are cargo-culting; yet more are fanning the flames of that cargo cult because their pay check depends on it.
Sysadmins are/were paid much less than cloud native devops people, and you need the same number of them unless you keep things very simple, which cloud providers do not incentivise. One need only look at AWS reference architectures.
You are certainly correct about the the overly complex AWS reference architectures. I've seen relatively simple applications with just as much infrastructure code (generally "terraform", occasionally CloudFormation JSON) as application code. It's crazy.
Most automation that you know of as “devops tools” are borne from sysadmins.
Terraform was written by a sysadmin; puppet, ansible, saltstack and cfengine too.
It is revisionist to think sysadmins were not automating their jobs.
Old school sysadmins used to know C, bash and Perl.
New school devops just traded that for a handful of DSLs and Python.
I’m sure some technicians working the IT department managed to get by running scripts created by sysadmins and claiming not to code, but it was definitely the common case that 20 years ago sysadmins could code and worked tirelessly to “automate themselves out of a job” (literally a mantra i was told as a sysadmin 15 years ago)
The people who wrote tools like terraform and ansible were engineers with system administration skills. Those are very rare.
The pay increased, the titles and tools changed but the mentality didn’t.
People just started beating their chest about devops and pooping on the legacy of sysadmins which is what most devops/SRE are.
Sounds like they were automating though.
“Cattle not pets” was a sysadmin mantra, but the business wanted pets most of the time.
I’m with you on most of your post, but this is straight up wrong.
https://www.linkedin.com/in/mitchellh/
Or are you saying that he's not the author? Commits seem to suggest otherwise.
commit fb9c58f0e20a85a5358142ddb39a2f74d3bbe69b
Author: Mitchell Hashimoto <[email protected]>
Date: Fri May 23 11:03:38 2014 -0700
config: better error message
commit 089822a36f0a5e4c583ab401b7496b48bfa31d65
Author: Mitchell Hashimoto <[email protected]>
Date: Fri May 23 10:52:19 2014 -0700
config: some comments
commit ec3f72703c82f50787ef86498d856a5798900ad8
Author: Mitchell Hashimoto <[email protected]>
Date: Thu May 22 16:56:28 2014 -0700
Initial work on config
commit 649cf336e86da10910f6e50ea3cfaad507a2e336
Author: Mitchell Hashimoto <[email protected]>
Date: Wed May 21 16:28:53 2014 -0700
Initial commitAn operations engineer and a sysadmin are two entirely different roles.
A system administrator administers systems while an operations engineer dedicated their time writing code to automate processes.
Sysadmins automate their work. Operations Engineer is just another title for sysadmin.
Systems Engineer is also another title for the same thing.
Platform Engineering (depending on where you are) is yet another variation.
If you're responsible for the operating system, monitoring, clustering, hardware etc; then you're firmly in "sysadmin" land.
Pedantry is requisite.
So do janitors. What's your point?
> Operations Engineer is just another title for sysadmin.
It really isn't. That makes as much sense as claiming that DevOps is just a sysadmin that's also a developer.
Many "operations engineers" are actually sysadmins. Many "devops" are really sysadmins also doing development.
That's not true of any place I experienced in the early to late 90s. If you meant earlier, perhaps, I wasn't there.
The growth of perl, for example, was in great part from the sysadmin community automating everything.
Remember that in those days you likely only had one machine. By "you" I mean the whole department. Everyone was logged into it and it handled email, talk, documents, compilation and debugging, etc.
That doesn't take away from the fact that the sysadmins were automating everything they needed to do, in the enviornment that existed.
The idea that I would need to automate the installation of an OS and applications on a fleet of machines was never contemplated because it didn’t make sense. I had one machine and OS upgrades arrived in the post every 6 months or so - on cartridge tape.
It was about need, not competence.
I worked in the mid-00s and it wasn’t a lot better in the dev space. People passing USB keys to each other was pretty common, SVN and CVS were around but there was a lot of developer code outside of it and peer review was going to someone’s desk and walking through the new code, nothing at all like what we have today for processing change requests.
You’re talking about systems administration when it was going from pets to cattle, the primordial period where people were automating but not applying software development practices on themselves yet because even the software development practices weren’t well defined.
Sysadmin was always lagging 5y behind development w.r.t. programming practises. The same is true today of devops.
tell me how many terraform repositories have unit tests or infra scripts for that matter which would be much simpler.
I did know some fantastic sysadmins at a local university around that time period. They had huge labs of Alphas and Suns and automated everything! Unfortunately, those folks didn't go into industry.
The closest I’ve seen to devops nirvana has been NixOS, but its frustratingly divergent terminology and file formats make it very hard to cross the chasm.
But nowadays if you want to manage your own metal then you can use those tools as well. You can still run hypervisors VMs on your own servers. You're just not paying Amazon 6x the price of what it would cost you, and you're not relegated to low-performing compute, memory, and IO elements that Amazon standardized on.
I’ve worked at exactly one storage company that had a customer that had exascale data. They were doing cancer research as I recall and their test machines generated a lot of data. I heard stories about CERN at conferences but they also self host their data.
But those were outliers. All of the large and small enterprises outside of that could fit all of their “big data” in the memory of a single blade server and still have plenty to spare. You can get machines these days with many TB’s of RAM.
I can tell you that massive amounts of storage are used by eg Mercedes Benz during simulations they need when developing cars. Back when I was reading about this - maybe a decade ago - they were using mind boggling amounts of storage. Still all before SSDs. You can assume all other world-class engineering firms in every kind of difficult engineering that requires physical robustness or safety have a similar need. But Mercedes Benz at least used an on-premises solution, simply because storing stuff on AWS would have created latency and throughput issues.
Is the issue big data or small machine?
Memory size of the largest AWS/GCP/... instances is a good indicator for small machine.
It doesn't mean I would go for the large machine. It just means I won't subscribe to "big data" as a reason to do X.
And it forces me to recalibrate this boundary regularly. Looks like double digit TB is currently the memory boundary for renting. I was still on the single digit TB train.
MinIO by itself is fantastic software for sure, but running storage appliances is shades of difficult or expensive.
On the one hand you can buy a netapp filer and support plan which is basically as self-healing as the cloud is, netapp will send a human to go replace failed drives before you even know a fault is imminent.
On the other, that's expensive but running your own is complex if you're not setup to do it already.
MinIO is but one part of a data storage puzzle. Though a very important one.
Try uploading 5PB of content and downloading it that day at 1M RPS.
Huge difference b/w reading a few pages of documentation on secrets manager, and clicking a button to get the service on vs deploying, and maintaining a vault on bare machines.
It's much easier to setup and forget basic bare metal servers with PG/NGINX and whatnot, than it is to automate using dozens of AWS services.
People pretend that AWS doesn't cost engineers to run it, when it's IMO basically the same human cost, if not bigger as complexity grows. You just don't pay that cost upfront, but you sure do pay it later with interest.
You get stuff like HA but that's not free. You also now have to manage a new boatload of services, scripts, changing APIs, etc.
3 years fast forward with hundreds of customers, and we haven't had a need to hire someone full time to "manage" these services. This may/will change down the line, but I couldn't have built my business without one of these cloud vendors.
Edit: I also know the pain of building (and managing) this on bare metal (last company did that) - and it was just apache/php/mysql on bare metal. It was a mess
This is a reference to the country Brazil [0], which centers on...
This is a reference to "the rule of the desk" [0], and "an elaborate, confusing structure", which centers on a monster [1], in this case a bureaucrat at a desk
[0] https://www.workhuman.com/blog/a-brief-history-of-bureaucrac...
I guess you've never done it yourself?
I'm not sure what those 4 engineers would be doing. You purchase a few servers with 5 year on site warranty and remote management, you take a couple of days to install them in some racks, and you never visit the site for 5 years. If they break, the manufacturer sends someone to fix them on site. The rest of the time you administer them remotely - just like you would AWS. If you want VM's install proxmox.
Whether it's cheaper than renting bare metal in a data centre is an debatable - colo costs a small fortune where I live.
But they aren't comparing it to that. They are comparing it to locking themselves into a closed source system that need specialised expertise to run, versus administering an open source Linux system that even the people running EC2 should be very familiar with. Most of the stuff AWS provides has open source counterparts - hell a whole pile of it is just open source they wrap a proprietary API around and charge you for.
10 times sounds like a lot, I'm guessing it's really less. Even so I come form the camp that shakes his head in disbelief at what people will pay AWS for basic VM and storage services, dressed up in a fancy API and marketing.
I know *data centers* that run on a few naive 20 yos admins/technicans + 1-2 "engineers" and all of them combined do receive salary of $5-10k/month in east eu
One of the companies had a picture of one of their "datacenters." It was something like 10 racks in a moldy unfinished basement with visible water on the floor of what I'm guessing was a residential building. Maybe they had mopped the floor for the picture?
I thought it was strange they would put that picture on their site and not some GCI or something. I guess at least you know they are not lying about being a real place?
they've decent building, data center tier, so power/network redundancy, etc, etc
they're just cheap as fuck when it comes to people
I really should start a blog for some of the weird research I do sometimes.
I'd like to remind everyone about Uber's experience: no EC2-like functionality until at least 2018, probably even now. Teams would negotiate with CTO for more machines. Uber's container-based solution didn't support persistent volumes for years. Uber's distributed database was based on friendfeed's design and was notoriously harder to use than DynamoDB or Cassandra. Uber's engineers couldn't provision Cassandra instances via API. They had to fill in a 10-pager to justify their use cases. Uber's on-rack router broke back in 2017 and the networking team didn't know about it because their dashboard was not properly set up and what the funk is eBPF? Uber tried but failed to build anything even closer to S3. Uber's HDFS cluster was grossly inefficient and expensive. That is, Uber's productivity sucked because they didn't have the out-of-box flexibility offered by cloud.
Look at Stack Overflow's architecture which stands apart because it was never designed to work in cloud from the beginning: https://stackexchange.com/performance
I'd argue that 90% of the SaaS doesn't have SO's scale. The whole thing would work just fine on a couple of FreeBSD servers running postgres and un-dockerized monolith. Half a rack at most with redundancy and replication.
But, if you've built your whole company around proprietary lamda functions and a vast range of AWS offerings, you're setting up yourself to never get out of the mess.
And you need dev servers. And a way to keep them in sync with production because you can’t just create a branch and provision a bare metal server on the fly. And you need a deployment process to get code from dev through test, UAT and production, and this process is lengthy and fraught because you can’t just fire up a new instance and switch the elastic IP, you actually have to deploy code on the physical machine and make sure there aren’t any configuration differences between your test and prod environments that’ll cause ‘it worked before’ problems. Developer productivity tanks, people are afraid to make changes, any pretence of agile/devops goes out the window and eventually everyone gets sick of the crusty old server and decides a complete rewrite is needed.
While Stack Overflow is popular in the dev world, not so much outside of it. The show they’re only serving 300 req/s. 19 servers to power the entire stack as well. I wouldn’t call this efficient.
At what point do you have the time/money/confidence to invest goodness knows how much in a data centre with space to grow, to purchase an enormous amount of capital to have it all installed etc. the building alone could eat that first years saving easily.
How many people are now needed to fault-find bad hardware/software/networks, to be on call for any problems? How many calls out to the Electrician to fix some power issue?
How much to setup and run a large air-con system for the data centre. Maybe not much in the US where aircon is common but much more expensive in Europe.
The fact they could afford to do this over such a short time period speaks to having a decent amount of cash on-hand.
Co-locating has no capital investment other than hardware, and is pretty cheap.
A 40U rack of compute charged as equivalent ec2 instances has a retail price easily of hundreds of thousands, if not a million+ USD per year.
Suppose each U has a $10k capital cost to make the numbers round, that is $400k in capital.
All this to say is that I don’t think capital is as big a factor as you might think.
Understanding your licensing and warranties is another huge cost that people don't take into account. We used to spend hours and hours figuring out if we could replace systems and what it would cost us.
Finally, you have to dispose of all that hardware when it gets too old or out of warranty. If you've never had to do that you probably have no idea how hard it is to do it correctly and so that it satisfies your SOC2 auditor.
All of these things (plus more) really add up. It's not just purchasing the components and installing them in racks. The management is probably even more expensive than the hardware.
And all of these problems go away with a cloud provider.
HDD replacement is trivial and some COLOs can even do it for you - and any good vendor will have a thing setup where your appliance automatically notifies them and a new drive is overnighted to your COLO. Sure - maybe a few times a year you drive out to swap a drive.
If you do not have good monitoring setup then that is entirely fixable - there are many stellar solutions these days out there. Hardware is easier than ever.
> large air-con system
You wouldn't usually jump from AWS to buying up real estate to build your own physical data center.
A sensible first step is to rent a rack at a colocation facility. They handle power, cooling, redundancy, physical access for you.
Don't underestimate the savings that can be made from switching from a big-name cloud provider to a more old school hosting provider.
At work, people keep complaining about our costs and coming up with spreadsheets showing how much money we would save with our own hardware.
They never add the engineering costs. When they do, they forget to include the ongoing maintenance. Or the new SMEs that need to be hired (and on call). Or even the opportunity cost of doing a multi-year migration to arrive at the exact spot they already are today.
All that money, and noone is looking into optimizing our systems to shrink the bill...
which is less than 80k, yes.
> You don't mention patching systems, or the time spend replacing the hardware racks every x years. It is very easy to underestimate the cost of maintenance.
Please stop assuming stupid shit, ask for clarification if you're not sure what I included in it.
Racking 2 racks took us 2 days, one on site, on on configuration.
Patching is also included in that 10%. We track that time. If you're patching manually you already fucked up, cloud or no cloud. Only time spent actually patching is "difficult" customers that need paperwork for taking down even one server of HA pair.
And it's utterly ignorant to assume cloud somehow saves you on this point, you just have more time used dealing on security stuff on dev side instead of ops side. Hell, I dare to say average VM with auto-updates will be more secure than average container deployment. Or maybe that's our devs that didn't get kicked enough to set their container pipeline correctly...
Yep, we should have pointed out that this has to be done in at the right time. Generally this article was about how we saved $1M expense and we was able to share this saving with our customers. Not an attack on AWS, I still use their platform for my other projects.
But to be accurate, this migration happened last year, and now we already doubled our usage and this brings us to $2M / year saving now.
I 100% agree with the comments below, use AWS while you are defining your product and growing, Todd (the founder) did this right. Spending days and nights on managing your own infrastructure or paying a devops employee to do it, in the early phase is just pure waste, and takes away focus. But we moved away from AWS at the right time.
Also, yes we had to do this to be competitive, as prerendering is a resource heavy service, and if your site is like ProductBoard (awesome tool) which will not consume terabits of traffic or use petabytes of ram, then you can stay on AWS forever and enjoy the benefit of not caring.
And what we lost? Minor inconvenience at this point, I cannot just pull up a new database in 10 minutes, but we don't really do that anymore, most of our current projects requires months of research, planning, and delivery. With those time frames we can notify our devops and get everything in place way before we need it.
So, keep on using AWS (it rocks!), just be sure to pay attention to the bill, and don't underestimate the cost of the traffic.
Have a nice one
$800k/yr is like the cost of 2-3 engineers. Even if they're capable of doing all of the work that used to have been done by aws, you don't have any room to expand without having to put up high capital costs, and certainly not on a dime.
It sounds amazing now, but wait till the future comes and you can no longer run your own data center as cheaply as amazon.
This company saved $800k/year. Perfect time to go in-house with this solution.
But when they were 1/10th this size, they'd only have saved $80k/year. Does that cover the cost of the engineering to build and maintain this system? Maybe not. And when they were 1/100th the size, it would have been laughable to go in-house.
At the right time, you make the right transitions.