I’ve _never_ worked with a “product manager” or “product owner” that could effectively tell me what a SLO was, or simply tell me what the business impact of bugs were. All I usually get is what we’ll lose if we don’t implement feature X or Y.
This impacts “DevOps” because it almost always puts focus and priority on anything feature-related. In order to do anything not feature-oriented, you literally have to be fighting massive quality problems on a daily basis. Thus... very few organizations understand how to have “devs and ops spend quality time together”. Almost all time not building features is wasted in their eyes.
In the end, this is why there’s a “dev and ops split”. The devs drive features, which PMs understand, and the ops makes things run, and the PO/PM types ignore them but sometimes like to make requests.
I think a pre-cursor to any real DevOps culture are SLOs that are understood and reported to executives. If that doesn’t exist, DevOps usually ends up being a BS term used to “remain relevant”. Without some kind of business quality objective, real DevOps time is pure cost, and counter to business development.
Just like “agile” BS, any real change usually starts at the top fo the company.
I can see the same when programming language advocates extol the benefits of more static analysis for software that does not even remotely need the level of reliability they are aiming for.
The guy who hacks it together in PHP will just get to market first and get such a critical mass of market share that 5-10% more reliability probably won't matter.
Of course a lot of this depends on market sector, self driving cars are a million miles from serving banner ads.
It kind of adds more credence to people not being so siloed in responsibility. We all just bring different skills to the party. Thus, we're all responsible for quality, for business impact and prioritization, etc.
Unfortunately, I've often had situations where, upon a reorganization, I'm left with managers who love to "guard others from distractions" and end up cutting team members off from broader discussions. Thus, leading to the siloing.
This is why I think "DevOps" ends up meaning nothing. It's a _lot_ of details that requires lots of communication and knowledge sharing that can be overwhelming to, ahem, certain PHB types. Sometimes you can implement a tool, other times you can just not build a particular feature, but the reasons are usually very complex and never easily get broken into a simple one-liner task definition with a story point cost.
Every bug or downtime that gets reported does have a visible business impact.
And unnecessary features seldom get implemented if not covered by project budget.
I have 25 years of engineering experience. I'm now a CEO... I think the cloud infrastructure movement is really killing off a valuable skillset - actually understanding your software stack.
My company would die on AWS... our hosting costs would be 8x...
And I agree with you in that one critical aspect, and would like to contribute my viewpoint: Every single successful move to the cloud I’ve witnessed hinged on the people who did it _really_ understanding their software stack.
For them (usually clients and not contractors or integrators doing the move for them) the key aspect of the learning curve was understanding cost drivers in cloud infra (or PaaS), weighing them against their current situation, and _measuring_ alternatives.
I’m often hassled by peers and salespeople for spending “too long talking to the techs” rather than doing pretty PowerPoint pitches, but I’m proud of the engineering work that makes those migrations viable, because a) it is truly full-stack and b) I learn at least as much as the techs I work with as we drill down into their stack and figure out how best to move it to the cloud.
Alas, this kind of depth work happens too seldom (see my other comment).
$171.83/month of AWS EC2 (+ bw): 8 core CPU 16GB RAM 250 GB SSD https://calculator.s3.amazonaws.com/index.html
24.99€/month of dedicated: 8 core CPU 16GB DDR3 250 GB SSD Basic Unmetered 1 Gbit/sec https://www.online.net/fr/serveurs-dedies/start-2-m-ssd
$809,57/month 20 Core EC2 dedicated instance RAM? Disk? BW? https://aws.amazon.com/fr/ec2/dedicated-hosts/pricing/
234,99€ month dedicated: 2x Intel® Xeon® Silver 4114 20C/40T 128 Go DDR4 ECC 2x 1 To NVMe Premium 1 Gbit/s https://www.online.net/fr/serveurs-dedies/core-5-s
I suppose the rest to 8x is on bandwidth, which is not charged on this kind of servers.
Provisioning an EC2 instance or a dedicated server just has the same cost in terms of manhour: running some automation in deployment pipeline, is something that you'll have to setup and maintain.
Be it updating a docker-compose file over ssh with ansible or something else: you have to do it anyway. If you want to cut that cost perhaps try with a starter project from OpenShift or Heroku, but that will also have a bigger cost even than an EC2 instance, and you will still have to take time to ensure your deployment is properly automated, with automated backups as part of the process.
It's always at least two in production if you ever need it to be HA.
Staging just needs n > 1 to test that sharding works at all.
[1] https://forrestbrazeal.com/2019/02/18/cloud-irregular-iam-is...
If it's throwaway then sure, but building your critical infrastructure around a single provider is really bad business hygiene for the sole reason that you can never control for your dependency increasing costs arbitrarily.
Maybe that's fine for most people, people seem to optimise (or value?) developer time over basically anything else so maybe it's something that's already taken into account.
Daily snapshots? For the prices listed you could have an actual hot spare running 24x7 and still save money. This also makes surge-up easier and if you ask nicely a lot of decent dedicated hosting providers will let you shrink without holding you to contract terms.
And even most dedicated servers can optionally come preinstalled with the OS.
to understand what devops should have been we have to step back and think about ownership. When you own they thing you build from you know building it, to deploying it, maintaininging and sunsetting it that's when you can tell me you are doing devops. It's not about a super developer or a super sysadmin - it's about full ownership. In the process you'll do software development, you'll learn about the environment and the hardware the software runs on and most importantly you will learn to never ever ever again overengineer something.
to answer parent: sysadmins have to learn how to write code. if you want to stay relevant and still have a job in 5 years you have no choice. AWS & other big clouds are already eating your lunch. the best transition is to actually learn how to deploy infrastructure in the cloud - and if you're starting fresh, why not actually learn how to code? good developers know their CI/CD pipeline in depth. They care about infrastructure and details. Apart from junior developers, you really don't have the option to live in your box and not care.
DevOps generally means developers doing sub-standard operations work.
Mainly because ops cannot be bothered to.
Telling me something is going to take a month when it should take only a few minutes if you’d done your job right the first time means I’m going to be doing substandard operations work.
But if that’s the case, is it really sub-standard?
It’s easy to pretend that, like say, HR, because you don’t see the work that’s being done, no work is actually being done, and you could do a better job if you’d had the job from the beginning.
However, how often do you get to do greenfield work as a developer?
Right, now imagine you’re not even writing code, just pushing configuration around. OPs is often a constraint satisfaction problem, not an engineering problem.
Developers can come along and whip up a quick ldap/logging/monitoring/pipeline, but when it gets to the long tail of the hundreds of hours of standard practice (ie. you have to write documentation for that thing you wrote) and additional tooling (yes, you do need to write custom integration to active directory because of [corporate policy here]...) it’s a big mess, badly done, by developers who didn’t really want to in the first place, and don’t want to maintain it (and sure as hell not out of hours when it goes down).
TLDR; don’t be ridiculous. Smart people do this stuff professionally and then you come along and think you’re some hot shot who can do it in a few minutes?
Get off your high horse. You have a domain skill set that doesn’t include competence in the domain you superficially think you could easily do.
Try listening to why people say it takes a long time, and work within the constraints of the domain, or you will produce substandard outcomes.
I’ve seen it in so many places, it makes me just sigh now. As a developer (which I am), my complaint with ”Devops” is that management has been given the same expectation you just articulated.
ie. that ops is basically people who were never good enough to be programmers, and basically, despite years of training and experience, are basically incompetent and not “agile”.
...and somehow, developers, with no training in the domain, will just do a better job of it.
They don’t. In any of the places I’ve worked and seen it tried, in like, the last ten years.
Ymmv.
only build for the next scale factor.
Basically, yes. Mainly because I do this professionally too.
Just because I haven’t been assigned to do it this time does not mean I can’t.
How about you are generous and interpret my comment as if I know what I’m talking about, then I won’t be an ass and nuance my previous statement a bit.
Something can (and probably should) take a month the first time around. Then, if you’ve actually fulfilled your promise as ‘devops’ any second time you are doing this thing it should take significantly less time. Maybe you’ve not automated all parts, but certainly the most common ones.
I think we’re now at iteration 20+, and it still takes a month for every new environment (dev, sqa, prod) of the same application.
Now it’s definitely possible there is a reasonable explanation for that, but it’s starting to get a bit suspect. Enough so that I’ll make a frustrated comment about it on HN when the subject comes up.
It could be a matter of work in progress or internal bottlenecks.
[0] has a good summary of WIP after the heading titled "Why Do We Need To Visualize IT Work And Control WIP?".
TLDR - read the Phoenix Project.
[0] https://itrevolution.com/resource-guide-for-the-phoenix-proj...
Then they're not great engineers. That's fine at the beginning, but in a long run, they need to care or they'll cause issues. The separation you mentioned and the possibility of creating higher order tools exists only if software can support that kind of environment.
Whether the software can be set up with a single file / endpoint / environment, or does it require interaction with a custom gui for initial setup for example defines whether ci/cd setup is trivial or a long project on its own. Unless devs can work to make things easy to automate, high order tools will not help them.
What kinds of issues do you think will happen? Isn't the whole point of having devops/sysadmins so software engineers don't have to spend valuable time setting up and maintaining infrastructure and tools? If the answer to that is no, then we've come full circle because software engineers maintaining infrastructure and tools are just devops.
Yes, and when they do learn how to code, you get stuff like Puppet.
Personally a DevOps/SRE type role appeals to me more because
1: I really liked ops but found coding and doing NOTHING but it kind of meh. A combo is fine.
2: I REALLY like approaching ops as a software problem
3: The work itself has more variety than straight ops or dev which I think is more suited to my adhd issues.
I think there is much political inertia and a lack of code savvy workforce keeping ops going but honestly the way I see it the writing is on the wall for traditional ops. I never see any ops positions in FAANG.
It may be true most coder s don't think about infrastructure but that gives more value to this niche not less. There is still value to DIY when it comes to things like infrastructure.
No more easily than they can become developers. There are a hell of a lot of really crappy cybersec guys around who think the job is just the access control subset of sysadmin...
1. There is a huge sysadmin workforce. Many of them never learned to really code. Are these people supposed to go away? Are they supposed to learn to code? Coding is a skill that takes time to master, and if these people were sufficiently motivated to code, they probably would've become coders before now.
2. In general, software engineers aren't thinking about or care about their CI/CD pipeline beyond it being easy to work with. They don't care about their infrastructure except insofar as what they need to know to make their stuff fast. There is a nice separation of concerns there that's nearly impossible to avoid. As a result, it's very easy to build higher order tools to handle most workflows and infrastructure. It's the whole reason AWS can exist and why most devs don't care about the details of ops.
I realize I'm speaking in very broad terms. However, broad terms are what define an industry. I've seen this play out enough times that I knew what this article was going to be about before I even clicked on it. I'm pretty sure most other people did, too.