That last sentence was something you never said to a boss back in '07-08. You would always know what was wrong and how bad it was within a few minutes of the issue, at least 90%+ of the time anyway. You could probably even guess how long it would be down for within ~10-15 minutes of discovering the issue.
FWIW, I think small/medium size businesses were/are probably the most affected by cloudification since those had the "most to gain" in the beginning. Then again, those were the types of businesses that I had the most interaction with until 2014-ish, I don't know what it looked like from a big-business perspective.
Cloud is to the IT department as big consulting companies are to HR.
And mainstream news will be either "company X offline for hours" if its just you, or "Amazon cloud outage takes multiple companies off line for hours" - the latter is better.
Average person hears on the news that company X or cloud Y had an outage, at most they have some anger towards the company for a day or so until the next thing in the news cycle comes along.
Switches are easy and cheap to put in a fail-over config. DB's are also running atleast in a primary / standby config everywhere. Besides if your running a single DB instance in a cloud VM the same thing can happen.
And don't get me started on the trouble you can get in with for example GCP SaaS PostgreSQL...
While I do look back with nostalgia on the days of editing files in Notepad++, FTPing them to a live server, then quickly switching to my browser to refresh the page and make sure it works...
Those days are in the past for a reason. No, git is not "fancy FTP". Yes, version control really does add much-needed structure to collaborative projects and change tracking.
You can still run a small business web server on a colocated server and FTP files to it if you want. You just won't find any modern engineers who want to touch that with a 10ft pole. The world has moved forward since then, and for good reason.
If you just add local machine testing, and a test server to that mix, what is the problem?
I was specifically referencing an old fad (might still be a thing but I haven't run into it in years) where production is a remote and a dev pushes their main branch to it to deploy. Yes, version control is amazing and it was something I used a bunch before my hiatus, but back then, it was usually SVN IIRC.
I think you're clinging a bit too heavily to the "good old days". It's ironic how this entire thread has turned into a mix of people either
1) complaining that modern technology is too complex and complicated
or
2) complaining that modern job applicants have lost the skills necessary to run things
There is much I love about today and I look forward to the future. I was just stating my observations. And yes, `git deploy prod` is a fancy FTP, but probably even worse than FTP if the build actually happens on the production server (which is something I saw more than once) and causes a production outage (also saw more than once).
Complexity is not a virtue. It is a dangerous resilience anti-pattern and should not be lionised.
Unless you're insinuating that things in fact remain simple but the complaints of complexity are down to the stupidity of commenters.
Its only innovative in an 'Apple released it and now everyone is on the bandwagon' way.
I've worked at companies that were using CVS for config file management in the 90s, the concept was already in play, it just wasn't as mainstream and didn't have a trendy infrastructure as code branding.
Good reasons for some people. But it turned out that those reasons (which you avoid explicating) were not so good for some people after all. Methods upon which you now "look back with nostalgia" had value which endures, namely control, simplicity, low cost at small scale, and a basis in standardised, accessible knowledge. These things are not valued much any more, but it's my opinion (and of others) that is a mistake.
I lived through part of that transition - I'll try and explicate through my own experiences.
The biggest thing automated deployments solved was human error. No more "oops, you forgot to FTP the JPEGs in binary mode" or "accidentally unploaded into a nested www/www folder". The transition was initially to tools like Ansible and Fabric which automated the transfer steps but had to be triggered manually. This also saved me a lot of time and pain.
However, this still left a gap where it was still possible to push broken code into production, so Continuous Integration (and/or Continuous Deployment) became fashionable. The benefits of these were you'd never push code that fails tests to production, the depth and breadth of tests being left up to you.
Incidentally, these developments coincided with my working with ever growing team sizes. There's no way I could have worked with 5 people with everyone S/FTP'ng stuff to prod. Neither would have Fabric + git been sufficient for four 5-engineer team working on the same software product. That's yet another benefit of the new way of doing things: it scales.
I think the additional benefits of checking-in your configuration and deployment steps into a version control system are self-evident.
Modern IT is swapped with CYA / No Accountablity
Cloud is not the only area this presents itself, everyone is more concerns about not being blamed for an issue than they are with resolving the issue. The more vendors you can point the blame to the better in their eyes...
AWS has never gone down across all regions at once. So we can do something about going down. You build the system to be multi-region and be able to evacuate traffic from one region to another, so when us-east-1 goes down again, you simply route around the damage and use resources in us-west2 until us-east-1 comes back up. People don't appreciate how much backend engineering goes into keeping something like Twitter up at all times.
Put it another way. Little guys go down when us-east-1 goes down. The big boys stay up no matter what.
Look back at all the big AWS outages of recent memory. Tons of the so-called big boys were down for the day along with AWS.
I still recall last year's AWS LSE when AWS went down globally, due to us-east-1
https://www.datacenterdynamics.com/en/news/aws-us-east-1-out...
There are famous examples of large media companies screwing over their internal IT depts through out the 2000's and 2010's
Are you really suggesting that search engine providers are manipulating their own search results to bury some sort of hidden "truth" about a people leaving the cloud? Suggesting that people need to "dig deep" to find the real scale of the issue sounds a lot like the "do your own research" line that gets attached to dubious conspiracy theories.
The simple explanation is that your student simply anchored her personal beliefs to an idea that this was a large movement, but her light research didn't match her pre-conceived beliefs. This is how conspiracy theories are born.
Regardless, this claim doesn't hold up to even basic scrutiny. I just Googled for "cloud repatriation" and all of the results are think pieces about why moving out of the cloud is a good thing.
Try it for yourself: https://www.google.com/search?q=%22cloud+repatriation%22&oq=...
> A good example was the broadcast industry. Entire branches of that industry doing post-production, colour, transcoding, and whatnot moved it all out to AWS. After price hikes, they wanted to go back to running their own services. But they can't. Because nobody knows about how to set-up and run that any longer - especially specialist things like buffers and transcoders. I mean, try finding a sys-admin who can just do simple tasks like set up and properly configure a mail server these days.
If you can't find sysadmins with basic competencies or engineers who can build systems as simple as buffering and transcoding, I can almost guarantee that the real issue is that you're not paying enough for the position.
Either that, or the applicants can recognize from a mile away that your company is not the kind of place that skilled engineers actually want to work for.
Don't mistake your lack of candidates for a lack of skill in the market.
If the average engineer doesn't have a certain skill, I think it's fair to say that it's hard to hire for it, because it will be for the average company.
"Don't move to the cloud!" is hard to make money by selling to places not in the cloud.
"Move out of the cloud" is harder to sell to places in the cloud than "move to the cloud" is to sell to places not in the cloud, because (a) bigger orgs with big budgets to make money off of tend to be older and not in the cloud fully in the first place and (b) moving out of the cloud is less turn-key - every client is going to be different and likely lower-margin than pushing clients into the same set of existing packaged cloud services.
That said, I'm not aware of any companies from people in my network that are actively moving out of the cloud... so I also don't buy that it's a big trend.
I don't have any knowledge of the particular topic, but you're suggesting that firms maximizing profit is a lunatic conspiracy theory. May not be what you intended, but your toxic comment was a bit nutty.
I should note that these warnings aren't accompanied by much sympathy; companies and academic institutions both do a profoundly terrible job at incubating the skills they need, then bleat at their local governments about new graduates not having the skills to be employable.
Today, Intel (and probably also AMD et al.) offer them exclusive deals on and even exclusive models of their hardware, due to the insane volumes "hyper-scalers" procure. In parallel, Amazon offers their own CPU arch (Graviton) cheaper than x86 cores to users of their platform in order to make it more attractive - at the insane margin AWS operates at, it can make these instances a loss leader, no problem. As a result, those in the x86 market that do not have a cloud business attached will grow weaker and weaker in comparison, and most cloud-native developer mindshare will shift to the cheaper option (since most engineering work in cloud envs is about reducing cost, period), which is AWS/GCP/... custom CPUs. And you, the random shmuck with an appetite for powerful on-prem hardware, will be left with worse products to buy for a higher price.
Also, cloud environments are rarely cost reductions. They are very frequently more expensive than on-prem deployments, but they are used because they are easy to spin up and down and they allow you to engineer things faster.
Anecdata: I have 2 customers who have had open roles of this type for more than a year each. I'm pleased to report that they are edging their salary offers up into "reasonable" territory - but the fact remains that someone with that skillset would ordinarily be able to get paid twice as much at any of the 3 major clouds.
Also noone is doing this then eventually (deskilling is real) and as I really don't want to work for any of the evil empires...
More likely that they're just not paying enough, and those skilled sysadmins would rather go to any other company that will pay them appropriately for their skills.
This is just another take on the "nobody wants to work anymore" rant that happens when companies can't understand why nobody is accepting their open positions.
Companies are not paying enough, but we also do not have as many opportunities (for SysAdmins) to learn things.
Paid Internships, Jr Admin Roles, etc are fewer and far between, companies are looking for "experienced" people in a time where alot of the experience is retiring.
Dont get me wrong there are plenty of us Gen X's out there with lots of experience but most of us are also massively over worked with limited opportunities given for us to train the next generation, so pay is only 1 part of this problem, head count is the other
Even if you find a company that has good pay, they almost always under size their IT dept by 25-50%, and rarely promote internal training or continuing education, or offer entry level admin positions with the goal to skill up people.
I think if you want to go back into history and look at things, we had sysadmins who were not paid enough, but fell into "give a man a little power" trap and showed a side of themselves that we would rather not see, fed and reinforced by getting blamed for anything bad that happened 'on their watch'. The cumulative generational trauma on both sides of that relationship sets us up for a situation where outsourcing IT to someone who can't sass back or harass us about our deployment plans seems like a good deal. Yeah it costs about the same and it'll interrupt our roadmap but I get to 'stick it' to the IT department. Fuck those guys.
Now it's been a while and people are starting to feel nostalgia for the old days.
It was never, has never, and could never be about "power". I have always been a generalist, with one foot in both roles of dev and admin.
I am very good dev in 3 programming languages (not including scripting languages like powershell and bash.. php, C#, and JS are my strengths), and passable in about 8 or 9 others (go, python, learning rust now, etc). I hold multiple certs and years of experience in sysadmin roles.
There are very few of us out there, people with extensive experience in both. I can tell you, once you have seen both sides of that wall you come way with the clear view that neither understand the other.
DevOps was proposed as a solution to that, and if Devops would have been implemented as a Management sytle not a role it might have accomplished that goal but sadly most orgs just eliminated the valuable perspective of sysadmins, and as you said "outsourcing IT to someone who can't sass back or harass us about our deployment plans" which from my experience many devs need to have their plans pushed back on because often time they are widely inefficient, insecure, ill thought out, and expensive
I have to wonder, if many of the cyber security incidents we have seen over the last 10 years could be avoided if Dev's lost the chip on their shoulder, and listened to the sysadmin more, instead of just rejecting their point of view as a "god complex" like you seem to
You know why all internet traffic goes over port 80 and port 443? Entire generations of admins who learned that their job was to say “no” instead of “how”. By the time the fingers got peeled away from that issue the damage had already cemented. Similarly ORMs also got a healthy nudge from DBAs who decided it was their job to protect the business from ever doing anything new. Today the priesthood of the DB is all but gone. Devs are expected to do those roles because otherwise nothing gets done.
And similarly, the Cloud doesn’t say “no”, it says “how much”.
Are bad things happening now? Bad things always happen. I read Applied Cryptography while I was still in college, and have worked at a lot of startups and in other small projects, so I am biased toward thinking devs should not be making other teams babysit them. Advice and control are very different things.
Now we have conflated network admins and sysadmin. Different things.
the reality is that was largely ISP's not sysadmins that caused that by throttling, shaping and in some cases out right blocking traffic.
Throw in some laziness by devs as well, it is widely inaccurate to lay that out the feet of sysadmins.
> Similarly ORMs also got a healthy nudge from DBAs who decided it was their job to protect the business from ever doing anything new.
Lots of ORM's were written by people that were DBA's because they were tired of devs that did not know how to construct a proper SQL Query, and thought that Select * was the always the correct way to return data.
This idea that devs had to ride in to save the industry from evil DBA attempting to block everything is a complete revision of history
>>my opinion plus decades of listening to devs vent
I am not sure how a circle jerk is proof of anything? devs in many cases may have gotten rid of sysadmins, but now a new beast is emerged that is making life hell for both sysadmins and devs alike. The "Cyber Security Analyst" and if the devs thought sysadmins saying no was a problem, you have not seen anything yet. Unlike the sysadmin cyberSecurity will be backed by compliance, insurance companies, and standards organizations. They will have actual teeth, and actual power something the sysadmin never had
Surely this calculation should include the cost of people with the required skills?
Something I've observed for some time as well. If Ops was truly outsourced to the cloud service providers then why is devops such a buzzword/trend, hmm?
And then they started taking away my Ops people.
Similarly, managing servers through Chef/Puppet used to be a thing just a decade ago. Now with the onset of serverless, Lambda etc., it's slowly fading away.
But you have to pay them enough or that is going to be like pulling teeth out.
And this is where the problems start.
A dubious conspiracy theory, frankly.
I know HN comments bias heavily toward anti-cloud, but the suggestion that all of the search engines are conspiring together to bury some massive underground anti-cloud movement (that conveniently confirms anti-cloud beliefs but is unfortunately just hidden by evil search engines) is laughable.
Still it's probably just the general noise of the fully oligopolized internet :).
That's the yardstick to use here.
I will do that. Horrible work. I would need >$200K/year....
Hell, for that matter, nobody knew how to set up a transcode server for live streaming 30 years ago - it's not some ancient art where we lost 50+ years of experience.
I don't believe that people who can learn to operate cloud services can't learn how to rack servers and install software. Ten to fifteen years ago I was in companies where ten percent of the software team was managing the datacenter. Now ten percent is managing kubernetes and AWS and all that. I've seen the cloud be better for smaller shops since you no longer have to have that server management if you only have one or two people, but I haven't seen it "deskill" things so much as "differently skill" them in larger orgs.
Was just thinking last night that I need to stand up a mail server to send/receive email from my self hosted products. Then I thought of the leg work involved and said words I’ve never though I would say: “I should just use SES and be done with it”
Who could have possibly predicted that the trap-shaped box was a trap?
The Trough Of Disillusionment is fast approaching. I'm surprised it's taken as long as it has. But then it always seems like the biggest, most expensive mistakes we make as an industry defend themselves, since nobody can bear to admit to themselves the magnitude of resources they set on fire in the process.
> As companies move their operations out to the cloud it's not the disappearance of hardware from the premises but the loss of skill-sets. Later they don't even know who to hire or how to write the JD to bring people back in.
So in that respect it's similar to outsourcing any other aspect of your business - there are many examples of hollowed-out suppliers of manufactured products that no longer have the ability to build things themselves.
They don't readily see that moving to the cloud increases entropy. You can't easily go back the other way, and so maybe companies should understand cloud as a one way process and think much more about the value they have and will be losing.
So that is where the next boom will be: a sysadmin-on-demand service. Quick, someone create a start-up which connects those with the required skills to those who need those skills, charge a 30% fee and become the next greatest thing since sliced bread.
Same thing with Jerusalem bugs. Wanted to check how careful I should be (I hike at night). First post a pest control company claiming that a bite can cause a fever. Further research into trusted sources mention no such thing.
I hate what Google has become. I miss the days where the first result in Google was valuable.
Pro-tip: if the post explains something and end with ‘conclusion:’ they’re pushing a product.
Moved to "devops" profiles since sys admins and regular ops guys stopped being valuable. Picked up puppet, ansible, aws/gcp. Moved to a big company, got bored out of my mind and decided to look for something else.
Questions from the first round of an interview(they call the round - * basics) in a small company - Advanced aws and k8s(no Linux or anything else in my resume). I cracked the round but was expecting it to include at least some Linux questions. Next was the scripting round which had nonsense questions involving a lot of calculation(because ops folks only know basic scripting). Cracked that too.
What I wanted to say is that there is no value in knowing Linux(to get a job, nothing beats knowledge on basics for debugging stuff once you land a job) and complex ops stuff since there is no market for these. Most of the people who were good at complex ops stuff moved into management roles seeing a lack of market for everything they knew.
I am planning to leetcode some more and move to a dev profile after a year or two since there is no upward career path for all the ops stuff. There is too much to prepare for devops interviews since there is no standardized interview template either(advanced Linux, advanced networking, advanced k8s, cloud providers, everything in my resume [a ton of stuff that I'm slowly removing one by one], ds and algos, leetcode].
It is not that there aren't people out there with the required knowledge, we are just removing all the seemingly useless stuff from our resumes.
Do you know what exactly increased in price? AWS usually doesn't increase prices, so I'd be interested if that was an exception or if that's a reason just given to cover another issue they had with AWS.
That's the case for a lot of different things, ranging from tech to politics to pretty much anything, and doubly so if people are selling something or otherwise have a strong interest in pushing something. It's just so much easier to extol the virtues of something rather than point out the problems with something, especially when the problems are not obvious from "common sense" at a glance.
"Searching for something" is often a very biased way of finding out about many things; it's not intentional, it's just that the "pro" and "anti" camps have vastly different levels of commitment to their cause, and you often need a lot of knowledge of context and nuances to make a fair judgement; any hack can say "X is brilliant!" but it's much harder to say "well, X is good in some cases, but [..]"
[1] https://files.ciokorea.com/archive/images/public-vs_-private...
.gif){kind=link}
Doing this correctly is mostly googling and using reference material to make sure you’ve set it up correctly, which means that anyone with decent Linux knowledge can do it.
I would not trust anyone who claimed they could set up a mail server with no external references.
She also noted that when you search on these terms the main engines (all run by cloud service providers) return poor results not congruent with the scale of the phenomenon. They are dominated by the opposite message obviously heavily SEO'd up to the top, plus shill pieces pushing cloud services but presented as "critical". Dig deep if you want to find the real scale of the "anti-cloud" issues.
Her main conclusion was very interesting though. That the big issue is not finance, reliability or control - but de-skilling.
As companies move their operations out to the cloud it's not the disappearance of hardware from the premises but the loss of skill-sets. Later they don't even know who to hire or how to write the JD to bring people back in.
A good example was the broadcast industry. Entire branches of that industry doing post-production, colour, transcoding, and whatnot moved it all out to AWS. After price hikes, they wanted to go back to running their own services. But they can't. Because nobody knows about how to set-up and run that any longer - especially specialist things like buffers and transcoders. I mean, try finding a sys-admin who can just do simple tasks like set up and properly configure a mail server these days.