I really think engineers end up on-call because companies give short shrift to documentation. Maybe if you afforded time to document systems and processes, you could outsource on-call duties, or trust sysadmins to remediate without developer intervention.

Which is why developers need to be on call, otherwise they don't build operational support into apps (monitoring hooks, useful logging, etc)

At my company devops is primary on-call, with a developer(s) as backup, if devops can't fix the problem quickly, they escalate to the developer. If they can't reach the developer or the developer can't fix it, then the entire release is rolled back and then the developers bear the brunt of the release schedule changes.

Without developers in that loop, it's hard for devops to get the tools they need to diagnose and fix problems.

Sort of defeats the purpose of Dev ops, if you have a Dev ops team and a Dev team, doesn't it?

Dev Ops is focused more on operations (tooling around releases, monitoring, etc) as well as doing initial triage for system problems to figure out where the problem is, and if they can't fix it themselves, determine which engineering group can help resolve an issue. Most of the Devops staff does development, but more around operational needs, product features are implemented by Dev.

Dev Ops means different things to different companies.

That said, there's a need for a way to divide the industry between traditional ops people and those with the ability to handle modern infrastructure and develop custom tooling where needed. And for better or worse the industry has decided that "devops" is going to be the way to determine that. Now the problem is all of the fakers (both individuals and teams) adopting the moniker. So now we're back to square one and we actually need to evaluate people and teams based on their knowledge and skill.

Our devops team sits in on design and code reviews to help ensure that operational needs are met early on in the process, and we'll change code to fix bugs or support operational needs. We're not full developers and won't rearchitect entire systems, but we will do bug fixes when we can.

So now we're back to square one and we actually need to evaluate people and teams based on their knowledge and skill

Have we ever left that square?

You're right that every org defines devops differently but it's totally and unashamedly coopted from a movement which, despite being very generally defined, defines it radically differently. And that's not to say it's always a bad thing either. Sometimes it takes a title to help signal change and make an organization better which in the end accomplishes the goal of the movement.

And I agree, I don't think we ever left that square and we never will. But if prepending "dev" to "ops" wasn't at least somewhat effective in getting something out of people in decision making positions it wouldn't be tacked on to job titles and team names left and right. So at least some people think the marketing terms in a job title are a meaningful shortcut even if smart people like you know better.

But ultimately, the developer is the one that best understands how their code may break so he needs to instrument the code accordingly. And when the developer knows that giving the right information to Ops may avoid a 3am phone call, they have good incentive to do so.

On call is a deal breaker for my future job searches, and I am considering leaving the company I just joined because they sprung it on me while never mentioning it in the interview process.

On call is justified in my opinion only if someone will die if the problem isn't handled. Or if compensation is dramatically increased and agreed upon in advance.

What if the company will die if the problem isn't handled? Many companies (mine included) provide a service that needs to be highly available 24x7 (customers run automated tasks 24x7 against our service). If our site regularly went down for hours at night (or even for an entire weekend) because of a software problem that no one could fix because the developers weren't picking up the phone, we'd lose customers and eventually, the company would go out of business.

Even a 10 minute outage is a significant event and requires a full RCA for customers. We try hard to architect for high availability, but bugs do happen.

Note that for some developers, you're never going to be able to compensate 'on-call' hours appropriately due to their evaluation of the opportunity cost of their time.

I'm glad you brought this up - there's a quick conclusion to this conversation:

"On-call time compensation is part of your salary."

Salary shouldn't be a thing that a company can ever hide behind.

Labor laws really do need to cover the maximums that an employee can be expected to work. They should also make going above those maximums exponentially more expensive in 'bonus time'; and the accumulation rate shouldn't magically reset after time, but only after time back on normal/reduced work load and duration.

Also, while I'm on this subject, 'full time' work should really begin at more like 24 hours / week. Benefits for part time work should be pro-rated. (It should never be more cost effective to split a full time job in to part time jobs. That is defrauding the economy and making others pay for the costs of your labor.)

IANAL but I believe (in the US or in NYS at least) this would be grounds for quitting with full unemployment coverage (you are not usually eligible for unemployment benefits if you quit, exceptions include the basic nature/hours/wage of the work being changed without your consent, since that is philosophically similar to having your old job terminated and being offered a new, worse one)

Get any on call expectations in writing when taking an offer if you can, just like you'd expect salary in writing

In a market as friendly to software developers as we're currently in, the correct response to that is totally "great, consider this my two week's notice, I'll go to this other company across the road that pays the same salary but doesn't demand I have on-time call".

(No, the response to every problem with your work environment should be to get a new job, nor is that even a realistic possibility for many. The economic argument still stands.)

Then they should be paying huge bonuses to be on call, or simply hiring for multiple shifts. It's pretty easy.

And when the company grows large enough that they can support 24x7 engineering staffing, they usually do it by having an overseas team.

Why? What's so special about a tiny company that they get things for free? What's so hard about, "If you want something, then pay for it"?

When you're in a 100 or 1000 person company, then it's easier to have dedicated after-hours support staff (or staff working from multiple timezones around the world)

No one is saying that it should be "free", it's built into the salary - every company that I've worked at that's had an on-call is very clear about on-call rotations during the interview.

Except you're saying that small businesses should get to ignore that math, and get stuff for free.

" That company may have a team of 5 developers, how do you split those 5 developers across 3 8 hour shifts that cover 24 hours/day 7 days/week + holidays + vacations + sickdays to get 24x7x365 coverage without doing an on-call rotation."

You hire more people so that you can, or you don't make deals you can't afford. Honestly, why is this so hard to understand? Why do you feel so entitled to things you can't afford?

My company has 100+ engineers, all of whom do a on-call rotation, they are all aware of it when the sign up, so that on-call duty is included in their salary. The employee can evaluate for themselves whether or not they think the salary is sufficient to cover that, but as far as I know, we've never had a candidate turn down an offer due to the on-call requirement (though I don't know that they'd always tell us that's why)

But really, does any engineer join a small company (in the USA) without assuming that they'll be on-call after hours? Even in early stage companies that haven't launched a product yet, there's still after-hours support to be done to keep dev systems running, fixing broken builds, etc.

the only reason my team puts up with it is because we're 100% remote. they never leave their families, so that's the trade-off.

This sounds particular to your experience. There are definitely companies where on-call time is compensated.

So if its not obviously worth my time, I just wont' do it.

I've been told here that now developers can and should do everything, but it doesn't seem like a lot of developers want to be called in at 2am.

In my world, developers barely understand the programming language or framework they're using and are more interested in using buzzword foo rather than solving the problem at hand or dealing with actually running the systems that solve said problems.

As someone who does both dev and ops jobs, together and seperatly, I think your job will be safe for a very long time.

Oh, man. A thousand times this. Many devs I work with have been writing code that runs on AWS for years, and don't know what RDS means.

HN is an enormous echo chamber; Take it with a liberal amount of skepticism.