It is absolutely easier to use S3 than to create your own fast, highly available, infinitely scaling storage solution on your own metal. It requires more than zero knowledge / expertise to use S3, but far less than it would to implement and run yourself.
If you can accept that, then we already agree in principle. It's just matter of where the line is drawn for various services and use cases.
It’s hard to quantify how best you’ll be served but a lot of people are following the mantra of “nobody got fired for going AWS”.
It makes sense for some people, others are cargo-culting; yet more are fanning the flames of that cargo cult because their pay check depends on it.
Sysadmins are/were paid much less than cloud native devops people, and you need the same number of them unless you keep things very simple, which cloud providers do not incentivise. One need only look at AWS reference architectures.
You are certainly correct about the the overly complex AWS reference architectures. I've seen relatively simple applications with just as much infrastructure code (generally "terraform", occasionally CloudFormation JSON) as application code. It's crazy.
Most automation that you know of as “devops tools” are borne from sysadmins.
Terraform was written by a sysadmin; puppet, ansible, saltstack and cfengine too.
It is revisionist to think sysadmins were not automating their jobs.
Old school sysadmins used to know C, bash and Perl.
New school devops just traded that for a handful of DSLs and Python.
I’m sure some technicians working the IT department managed to get by running scripts created by sysadmins and claiming not to code, but it was definitely the common case that 20 years ago sysadmins could code and worked tirelessly to “automate themselves out of a job” (literally a mantra i was told as a sysadmin 15 years ago)
The people who wrote tools like terraform and ansible were engineers with system administration skills. Those are very rare.
The pay increased, the titles and tools changed but the mentality didn’t.
People just started beating their chest about devops and pooping on the legacy of sysadmins which is what most devops/SRE are.
Sounds like they were automating though.
“Cattle not pets” was a sysadmin mantra, but the business wanted pets most of the time.
I’m with you on most of your post, but this is straight up wrong.
https://www.linkedin.com/in/mitchellh/
Or are you saying that he's not the author? Commits seem to suggest otherwise.
commit fb9c58f0e20a85a5358142ddb39a2f74d3bbe69b
Author: Mitchell Hashimoto <[email protected]>
Date: Fri May 23 11:03:38 2014 -0700
config: better error message
commit 089822a36f0a5e4c583ab401b7496b48bfa31d65
Author: Mitchell Hashimoto <[email protected]>
Date: Fri May 23 10:52:19 2014 -0700
config: some comments
commit ec3f72703c82f50787ef86498d856a5798900ad8
Author: Mitchell Hashimoto <[email protected]>
Date: Thu May 22 16:56:28 2014 -0700
Initial work on config
commit 649cf336e86da10910f6e50ea3cfaad507a2e336
Author: Mitchell Hashimoto <[email protected]>
Date: Wed May 21 16:28:53 2014 -0700
Initial commitAn operations engineer and a sysadmin are two entirely different roles.
A system administrator administers systems while an operations engineer dedicated their time writing code to automate processes.
Sysadmins automate their work. Operations Engineer is just another title for sysadmin.
Systems Engineer is also another title for the same thing.
Platform Engineering (depending on where you are) is yet another variation.
If you're responsible for the operating system, monitoring, clustering, hardware etc; then you're firmly in "sysadmin" land.
Pedantry is requisite.
So do janitors. What's your point?
> Operations Engineer is just another title for sysadmin.
It really isn't. That makes as much sense as claiming that DevOps is just a sysadmin that's also a developer.
Many "operations engineers" are actually sysadmins. Many "devops" are really sysadmins also doing development.
That's not true of any place I experienced in the early to late 90s. If you meant earlier, perhaps, I wasn't there.
The growth of perl, for example, was in great part from the sysadmin community automating everything.
Remember that in those days you likely only had one machine. By "you" I mean the whole department. Everyone was logged into it and it handled email, talk, documents, compilation and debugging, etc.
That doesn't take away from the fact that the sysadmins were automating everything they needed to do, in the enviornment that existed.
The idea that I would need to automate the installation of an OS and applications on a fleet of machines was never contemplated because it didn’t make sense. I had one machine and OS upgrades arrived in the post every 6 months or so - on cartridge tape.
It was about need, not competence.
I worked in the mid-00s and it wasn’t a lot better in the dev space. People passing USB keys to each other was pretty common, SVN and CVS were around but there was a lot of developer code outside of it and peer review was going to someone’s desk and walking through the new code, nothing at all like what we have today for processing change requests.
You’re talking about systems administration when it was going from pets to cattle, the primordial period where people were automating but not applying software development practices on themselves yet because even the software development practices weren’t well defined.
Sysadmin was always lagging 5y behind development w.r.t. programming practises. The same is true today of devops.
tell me how many terraform repositories have unit tests or infra scripts for that matter which would be much simpler.
I did know some fantastic sysadmins at a local university around that time period. They had huge labs of Alphas and Suns and automated everything! Unfortunately, those folks didn't go into industry.
The closest I’ve seen to devops nirvana has been NixOS, but its frustratingly divergent terminology and file formats make it very hard to cross the chasm.
But nowadays if you want to manage your own metal then you can use those tools as well. You can still run hypervisors VMs on your own servers. You're just not paying Amazon 6x the price of what it would cost you, and you're not relegated to low-performing compute, memory, and IO elements that Amazon standardized on.
I’ve worked at exactly one storage company that had a customer that had exascale data. They were doing cancer research as I recall and their test machines generated a lot of data. I heard stories about CERN at conferences but they also self host their data.
But those were outliers. All of the large and small enterprises outside of that could fit all of their “big data” in the memory of a single blade server and still have plenty to spare. You can get machines these days with many TB’s of RAM.
I can tell you that massive amounts of storage are used by eg Mercedes Benz during simulations they need when developing cars. Back when I was reading about this - maybe a decade ago - they were using mind boggling amounts of storage. Still all before SSDs. You can assume all other world-class engineering firms in every kind of difficult engineering that requires physical robustness or safety have a similar need. But Mercedes Benz at least used an on-premises solution, simply because storing stuff on AWS would have created latency and throughput issues.
Is the issue big data or small machine?
Memory size of the largest AWS/GCP/... instances is a good indicator for small machine.
It doesn't mean I would go for the large machine. It just means I won't subscribe to "big data" as a reason to do X.
And it forces me to recalibrate this boundary regularly. Looks like double digit TB is currently the memory boundary for renting. I was still on the single digit TB train.
MinIO by itself is fantastic software for sure, but running storage appliances is shades of difficult or expensive.
On the one hand you can buy a netapp filer and support plan which is basically as self-healing as the cloud is, netapp will send a human to go replace failed drives before you even know a fault is imminent.
On the other, that's expensive but running your own is complex if you're not setup to do it already.
MinIO is but one part of a data storage puzzle. Though a very important one.
Try uploading 5PB of content and downloading it that day at 1M RPS.
Huge difference b/w reading a few pages of documentation on secrets manager, and clicking a button to get the service on vs deploying, and maintaining a vault on bare machines.
It's much easier to setup and forget basic bare metal servers with PG/NGINX and whatnot, than it is to automate using dozens of AWS services.
People pretend that AWS doesn't cost engineers to run it, when it's IMO basically the same human cost, if not bigger as complexity grows. You just don't pay that cost upfront, but you sure do pay it later with interest.
You get stuff like HA but that's not free. You also now have to manage a new boatload of services, scripts, changing APIs, etc.
3 years fast forward with hundreds of customers, and we haven't had a need to hire someone full time to "manage" these services. This may/will change down the line, but I couldn't have built my business without one of these cloud vendors.
Edit: I also know the pain of building (and managing) this on bare metal (last company did that) - and it was just apache/php/mysql on bare metal. It was a mess
This is a reference to the country Brazil [0], which centers on...
This is a reference to "the rule of the desk" [0], and "an elaborate, confusing structure", which centers on a monster [1], in this case a bureaucrat at a desk
[0] https://www.workhuman.com/blog/a-brief-history-of-bureaucrac...