I don’t follow this. Developers are responsible for learning what kind of environment their application runs in, but ops is not responsible for having some clue about what they’re running? That cuts both ways, and it’ll help everyone out.
> Developers are responsible for knowing if their stuff is going to fall outside of those, and come to us to work something out.
I find this attitude fairly common amongst ops people. They just build something that is totally inappropriate for actual usage, and then dump the responsibility for figuring that out on the developers.
I don't think it's as cut-and-dried as your question frames it, but I do think there are fundamental differences between the two positions that justify some of the tension there.
The problem is the difference between domain knowledge and general systems knowledge. The former varies wildly from org to org, team to team or even within individual teams. The latter is more consistent across wider applications and over longer timeframes.
Developers usually need a lot of domain knwoledge to do their job, which can leave less space for systems stuff. But the systems stuff they do learn tends to be more widely applicable.
Ops folk often service many teams where the domain knowledge differs between them. The best of them might be able to internalise all of those differences but it's a big ask. And there's rarely any crossover.
This difference is also why developers tend to have a slower ramp-up time than ops engineers do on joining a new team. It's just the nature of the work.
I say all this as someone from the developer side of the fence. I'm fortunate to have some years in the bank now that the systems stuff comes more easily. The domain stuff remains really hard.
With the road metaphor, one issue I've seen is ops will create a rope bridge and get mad when devs need to drive a car over it. "You shouldn't do that! You idiot! Just walk over the bridge like we expect!"
Example: We have about 500 different applications in our company and the ops team maintains a single rabbit cluster for all apps (and everyone is supposed to use that one cluster). If an app gets too chatty on that cluster "Oh you idiot, why are you so chatty! You just sunk the organization!" Which, in turn, discourages the usage of rabbit (maybe that's the intention?)
> But most of my team does not write code.
I actually prefer this ( :D ), our ops team was a bunch of converted devs that decided the best way to do things was making a giant ops framework for all devs to follow. That ended up costing WAY more money than if they'd just used tools that were available. They fetishized trying to make everything "just one line!" which ended up breaking anytime you had a slightly different need (trying to take control right up to managing how version bumps happen).
Overly trying to force a single method of implementation has a lot of negative consequences. I prefer instead to have guidebooks and examples with the freedom to be an idiot and walk off the beaten path when needed.
Well, the main problem with the "bridge mismatch" is usually that resources required for an environment are not free. Its usually the opposite, most infrastructure is rather expensive, and running multiple systems side by side because multiple developers require slightly different versions of the same thing tends to explode cost.
The highway road crew know what a car is, though, right? They know that the road needs to be clear and flat and drained of water, and the markings need to be clear, so that cars can drive on it.
When the devs come to you complaining about flat tires, you can't turn round and say 'this is a mechanic issue, I don't know how tires are meant to work. They go on the bottom, right?' - you're meant to help check for rusty nails or bits of metal in the road that are causing all these flats.
'Oh, I didn't realize that was something that could cause trouble for cars'
Well then you're a pretty crappy highway maintenance guy.
How? Honest Question. If you know nothing of the application how are you able to offer any input into the infrastructure it runs on.
You try to respond to what people need and add things when there is enough demand. But I can't know what your business goals are, what your uptime metrics are, or who your users are.
At some point, your app becomes a black box that takes in requests, accesses DB/storage, and emits logs/metrics. I just don't have the brain space to be intimately familiar with each service.
Do you... work for the same company? Draw your paycheck from the same revenue stream?
If you're just a black box provider of undifferentiated compute/storage why the heck are we paying you? We can buy that from a dozen cloud PaaS providers.
I do my best, but I'm never going to be intimately familiar with your product on a technical or business level in the way that you are when you spend 20-40 hours a week on it. If you want that level of service, you're gonna need another couple million a year in ops staffing budget.
You're paying ops because someone needs to know how to fit all the lego AWS gives you together, understand what's inside the lego pieces to debug issues when things go wrong, be accountable for ensuring best practices are implemented as far as security, backups, etc, optimize spend, and figure out how to architect all this stuff to make sense on AWS.
We could get some of that with some of the more managed services like Heroku, but at our scale the premium we'd pay is waaaaaay more than two ops salaries.
The more they need me, the bigger my paycheck gets. My TC has increase 4x last five years and I do not consider myself some infra guru by any means.
I agree with point 1, nearly completely. A lot of developers could take a lot more responsibility for understanding the environments their applications operate in, but I get it.
Point 3, at least in my shop, you're just wrong. I don't know anything about what you're writing. I probably don't even know what problem it is supposed to solve. You are mistaking the highway road crew for mechanics.
Point 4, in my shop, we provide a lot of documentation and guidelines for this sort of thing. Developers are responsible for knowing if their stuff is going to fall outside of those, and come to us to work something out. Again with the road metaphor, if you drive a semi into a single car garage, you're the idiot, not the person who built the garage.
On some of this, I'm taking a hard line. I do, in fact, end up doing a lot of troubleshooting with developers. But most of my team does not write code. If you want more senior ops folks who also have a coding background, come on over! There aren't that many of us who are any good, and I would love to hire more.