Dev folks often take weeks to make even a small change. It doesn't matter how urgent the need is or how badly the business needs some kind of workaround, they over-complicate every problem, and ops has to spend more time in meetings planning their next project than it would have taken for us to put a working fix to be in place. Maintainability is one of the biggest sins I've seen in devs, tools like npm encourage devs to use an overly-complex chain of 3rd-party dependencies for even small projects, and they rarely ever update their dependencies after initial deployment, so if a security update to an underlying package is required or if the underlying has to be deployed to a new underlying system or container changes, things can break easily. They depend on IDEs for everything, and can't even use commonly installed system tools like awk or sed when, or use netstat to debug simple networking issues. It's like the 1970s never even happened and they never even learned what an operating system is.
spoilers: it is not just a web dev thing
The issue is there’s no formal tool or practices for testing in ops. DSL languages are limited forcing you to use several for different kind of scenarios. At the end you need to rely in a real programming language to parse different format files and ensuring your variables are correct. I think developing software for operations is exciting which is going to mature with time. Kubernetes(and its CRD) is a step forward
Sure, things like ansible lack proper testing support, but that doesn't mean that all of the profession doesn't test.
There is though, it’s called writing normal software. Kubernetes can be framework to build on, with actual developers who design from high level logic down through to the implementation.
I’m not sure what job listings they’re looking at, but as an ops person (kubernetes) I’m interviewing for jobs with close to half a million a year in total comp.
Working as a data scientist - software engineer in a midsize company, I constantly battle amateur ops folk and "backend" fullstack jockeys from introducing kubernetes into a saas product I mostly created by myself (makes money but there are maybe ten users per hour tops, why would I need kubernetes for that?).
My org has seen multi-day downtimes for the entire eng team workflow because the eks cluster went down and they couldn't figure out how. We have four people dedicated in the infra team for this! I'm not really an ops person but I see where the failings of these folks and stacks are, and feel like I might be able to learn to be half-decent if I put the time. What advice would you give ?
I've found nomad to have a lower complexity than kubernetes but if you cannot directly integrate service discovery into your application then you will need to use a service mesh which is an all or nothing thing but using something like traefik's support for consul means you will have to use regular service discovery alongside the service mesh. It's not a huge burden but there should be a better way.
They need k8s for their, otherwise bleak CV. Its all hype and shiny today.
The idea is that you move your low traffic app onto servers with other low traffic apps and save money. If they’re just moving your app to a kubernetes cluster by itself it’s probably not worth it.
Are you talking to FAANG’s? Second tiers like Uber, Square, etc? Other?
"Oh that nil pointer exception on line 53? Yeah that means you don't have IAM permissions".
In short, the ops world lacks what most software engineers would consider basic engineering practices. It really feels like that entire world is just a hackathon project. I know there are some very talented engineers on ops teams. It's just the impression I got from 6ish month rotation.
Rant over, this just touched a nerve.
But since the CEO has a development background, always sides with them and in general wants them to build "features", not fix their shit, they never have to take responsibility for anything.
this is mind boggling! For med size company to have a separate ops department. I can't imagine that responsibility for MY code in production (and delivery of the said code to production) lies on a different team/department that are possibly not even on the same floor/building I'm at.
Separate OPS department is where dev's happiness and job satisfaction go to die.
I'd imagine this outdated setup in some government agency but not in a med-size company.
I recently did some interviews with our devs to find out what features we could add to our platform that would provide them with value. The result was interesting and I found that people typically fall into one of two camps: 1) I want to know everything about my deployed service and have tools that alert me and allow me to intervene, or 2) I don't want to know anything about the deployed service runtime and expect operations to handle my issues and alert me when there's a problem.
It sounds like you might be in the former. =)
Initial intention aside, it feels that there is a general consensus that VAST majority of the companies that have `devOps` in the job description somewhere are cargo culting and have no clue what they are doing.
If you slap `Dev` prefix to your Ops department your job postings will look "trendy" but nothing else will actually change.
DevOps is a culture, not a team or department.
None of those things are related to your code directly, but may interact with it at some level. At some point, you get so far removed from the work on your actual application that it makes sense to move that to another 'Infrastructure' group.
I would certainly see it as a departure from what I want to do - design, build and deliver systems. I'm not really in it for the continuity operations.
Owning (as in caring about and as in business responsibility) the reliability and operation of systems you build, at least for a while after they stabilize, is critical if you want to produce quality products. After all, operational flakiness is a UX issue.
Sorry but that's utter bollocks. I'm not interested in being in ops, therefore my software is shit?
> Owning (as in caring about and as in business responsibility) the reliability and operation of systems you build ... is critical.
But that's not being in ops. That's about taking an interest in the running system.
You've just jumped on me saying I'm not interested in having a role in ops because I like to build software, and run off to some weird unsupported conclusion that I just write code and abandon it. I don't need to have a business responsibility for the running system in order to support it and be responsive to the ongoing needs of those that do.
You are so lucky. This is incredibly rare. I'm also lucky that my team is the same way. But most teams are not.
> Cloud monitoring is a saturated market.
It's like yeah, it's saturated but it's saturated because the infrastructure is we're hosting it on is still and forever changing. Take some serverless CloudFormation in AWS; there was no good solution for application monitoring until someone specifically started solving for it because no one in their right mind was going to use CloudWatch and none of the other existing monitoring solutions/tools could fit the bill either unless they started from scratch and solved for that specific new infrastructure.
<Insert shameless Epsagon plug here />
The Cloud monitoring market might seem saturated but that's because there is no "silver bullet" solution given how much infrastructure has been and continues to change.
I'll never do business with a company that gross.
Code and tooling is built on a "what works" basis and no particular attention seems to be paid to the overall design of the software or testing (likely due to a real or perceived lack of time). On-call rotations (and how to eliminate work for them) is the hot button discussion topic.
The #1 question I get asked by the developers is "Did you really want to transfer from development to ops?" which me think that a lot of developers look down on operations roles and see it as a demotion. I find that quite odd given that 1) ops keeps the product up and the money coming in with relatively little headcount and 2) most of the people working on our ops teams have a formal education in software engineering.