News for hackers. Stuff that matters.
Re:The Ops Identity Crisis by llama052 on Saturday October 15, 2016 @12:41AM (#12712737)
I've always seen operations folks in a different light then most developers. Maybe I'm an exception but I've always viewed operations people as having in-depth knowledge of infrastructure on the lower layers (Session layer and lower). They are able to walk through the inner-workings of network concepts, operating systems, etc. They can implement and design systems that work in a highly available, scalable way.

Not saying developers CANT do this, but a majority of them don't want to, and don't have the first idea on how to. Granted some "10x" developers who have full stack knowledge can do just as well, but let's be honest most people aren't 10x developers, they want to mess with the code stack and that's it.

This reminds me of the all the job posting you see online that want you to have

          expert knowledge of C++
          expert in SQL DBA 
          expert in networking 
          expert in web design
          expert in big data
Yeah you might be able to find someone who can do all of the above, but odds are they won't do them all well.

I don't think the expectation should be set that developers should have to manage the systems stack, as well as manage the code.

Personally, I can't imagine having the expectation set that I need to be on call, and work on code commits with deadlines, while debugging networking issues in production.

Even with configuration toolsets, and infrastructure moving to "code-to-deploy" solutions. Things will still break, unusual things will still happen. Taking developers out of their zone to focus on problems will slow down the entire company.

Ops can can always push to 'strive to automate themselves out of their jobs' but I'd argue that this is an endless job which always has outlets that you can continue to strive and build into.

Of course, this is talking from my personal experience, I've never worked in a big company like Uber, so the environment might be entirely different from my own.

Re:The Ops Identity Crisis by donavanm on Saturday October 15, 2016 @05:23AM (#12713356)
> Personally, I can't imagine having the expectation set that I need to be on call, and work on code commits with deadlines, while debugging networking issues in production.

This seems to be the meme of "the supporting infrastructure always breaks and causes the pager engagements!" I have seen those orgs and times where networking or facility failures are the leading cause of outages. Long term those are symptoms of chronic underinvestment and tech debt accumulation.

I also see, on a daily basis, a massive business where the absolute leading cause of "outage minutes" is software defect or deficiency. With investment in supporting infrastructure you're paged 10 times for a defect your team "developed" for every one unavoidable dependency failure. Big companies can make those (huge) investments themselves. Small companies can pay someone else for access to theirs. In any case its always a business decision, not a certainty.

Re:The Ops Identity Crisis by davidgerard on Saturday October 15, 2016 @06:02AM (#12713440)
I explain my job to nongeeks as "computer roadie". My job is to make sure everything is in order, devs' job is to get up there and be Eric Clapton.

People can do both (many roadies are really quite capable musicians). But they're fundamentally different mindsets, and expertise in one is not transferable to the other.

Re:The Ops Identity Crisis by toast0 on Saturday October 15, 2016 @01:09AM (#12712814)
> Personally, I can't imagine having the expectation set that I need to be on call, and work on code commits with deadlines, while debugging networking issues in production.

"sorry about the deadline, I had to make the fine network work" gets you out of making deadlines.

The nice thing about being on call for your own code is that fixing your stuff at all hours is a natural consequence of writing fragile stuff. The not nice thing is that not all of the reasons your stuff breaks are your fault and some issues aren't realistically preventable.

Re:The Ops Identity Crisis by llama052 on Saturday October 15, 2016 @01:23AM (#12712850)
I can understand being on call for your own code, but often that's just not the case.

Infrastructure can go off the rails as well, clusters can go on the fritz, upstream might be having issues, a release update on your infrastructure side might introduce problems. I just don't see that falling on the developers who also must maintain the code-base for the product.

Re:The Ops Identity Crisis by pjscott on Saturday October 15, 2016 @02:15AM (#12712967)
That just means that being on-call for your own code results in good code, a love for automated testing, and a seething grudge against EC2 and its tendency to have multi-zone outages at 3 AM. It doesn't matter what time zone you're in, by the way; the outage will always, always page you at 3 AM.
Re:The Ops Identity Crisis by toast0 on Saturday October 15, 2016 @10:52AM (#12714103)
If the issues for things that aren't your fault happen are infrequent, then it's not a huge burden, and developers may be able to help with mitigation: ex if your internet bandwidth is cut in half because of a fiber cut, how can you move or shed load to have a stable, degraded system.

If the issues are frequent, then it provides incentive to build fault tolerance. (And also seething grudges for fault prone infrastructure)

Re:The Ops Identity Crisis by morgante on Saturday October 15, 2016 @12:56AM (#12712772)
> Personally, I can't imagine having the expectation set that I need to be on call, and work on code commits with deadlines, while debugging networking issues in production.

Really? As a dev, the vast majority of my projects/roles have required all 3 to some extent. I have been on call for the past 4 years.

Honestly, people really need to admit that the age of the sysadmin (who is not also a strong programmer) is over. I might not have enough deep knowledge to answer network/OS trivia off the top of my mind, [0] but I'm perfectly capable of running a production build for my code (with monitoring, configuration management, etc.) and finding appropriate resources in the instances where problems are outside my domain.

[0] https://news.ycombinator.com/item?id=12701272

Re:The Ops Identity Crisis by llama052 on Saturday October 15, 2016 @01:00AM (#12712781)
> Really? As a dev, the vast majority of my projects/roles have required all 3 to some extent.

I can run photoshop and mock some images at home and it's good enough, but does that make me a photoshop guru? I think not.

edit: bad example.

Re:The Ops Identity Crisis by morgante on Saturday October 15, 2016 @01:06AM (#12712808)
> To some extent maybe, but running something on a heroku engine

Talk about a straw man. I would agree with you that deploying to Heroku hardly constitutes ops. I'm talking about things like running a multi-AZ Kubernetes cluster.

Re:The Ops Identity Crisis by llama052 on Saturday October 15, 2016 @01:18AM (#12712835)
Following a guide online and getting something online is different from running something in production that a company relies on. Learning a infrastructure piece thoroughly can take months alone in resilience testing, failover testing, and system design/architecture choices.

It's a huge fallacy in the infrastructure side of things that because something "works" means it's setup correctly, and that couldn't be further from the truth.

You'll get an environment built on technical debt. Outages will occur and you'll be playing sysadmin full time before long. You won't be a developer at this point anymore, you'll be the sysadmin/ops role of your company.

I've seen this happen at multiple companies, and I doubt this is the last time I will see it happen.

Re:The Ops Identity Crisis by morgante on Saturday October 15, 2016 @01:24AM (#12712853)
You continue to be extremely insulting and making significant assumptions about my experience. What shibboleth do I have to provide to prove that I know what I'm doing? This inherent assumption that other people don't know anything is incredibly damaging to the industry: I don't assume a SRE is an incompetent programmer, so you shouldn't assume I am barely competent at ops.

> Following a guide online and getting something online is different from running something in production that a company relies on

Where did I say that I had simply "followed a guide online" instead of "putting something in production."

Like I said in my original comment, I have been the primary on call engineer for multiple projects with millions of users.

Despite that, I still find time to develop new features and push the product forward. Yes, I occasionally deal with outages/failures and have to write some new automation to remediate them. But that doesn't make me a full-time sysadmin.

I don't really feel like engaging with you further, since you're apparently more interested in arguing with a straw man of your own construction.

Re:The Ops Identity Crisis by llama052 on Saturday October 15, 2016 @01:36AM (#12712883)
> You continue to be extremely insulting and making significant assumptions about my experience. What shibboleth do I have to provide to prove that I know what I'm doing? This inherent assumption that other people don't know anything is incredibly damaging to the industry: I don't assume a SRE is an incompetent programmer, so you shouldn't assume I am barely competent at ops.

My mistake for projecting, the comment you posted about sysadmin age being dead gave me the opinion that you think infrastructure is just something you can deploy and forget, again my mistake.

I also spoke in my original message that some 10x engineers can do both, but they are not the majority.

I guess you're a 10x full stack engineer then.

> Like I said in my original comment, I have been the primary on call engineer for multiple projects with millions of users.

That's great, so you're a developer, and operations engineer then (devops)?

Do you think that developers should be expected to do both? Genuinely curious

Re:The Ops Identity Crisis by morgante on Saturday October 15, 2016 @01:43AM (#12712904)
> My mistake for projecting, the comment you posted about sysadmin age being dead gave me the opinion that you think infrastructure is just something you can deploy and forget, again my mistake.

I didn't mean to imply that. The "who is not also a strong programmer" part of my comment is important. Of course infrastructure breaks, but in the modern era we have the tooling that your next step should be figuring out how to automate things so that never happens again.

> Also looking at your linkedin on your profile led me to believe that you worked with small niche infrastructure setups that don't need to scale.

I'm not sure where you got that from. As much as people on HN hate Business Insider, it's hardly a "niche" site. And for the record it's not even the biggest site I've worked on.

> I also spoke in my original message that some 10x engineers can do both, but they are not the majority.

I agree that the majority of engineers can't do both, but I also don't think it's only small minority who can. Infrastructure is just another skill-set that some engineers can/do build.

> That's great, so you're a developer, and operations engineer then (devops)?

I wouldn't call myself an "operations engineer" but I've done quite a bit of devops work.

> Do you think that developers should be expected to do both? Genuinely curious

I don't think every engineer should be expected to do both, but I do think at least one developer on every team should have DevOps experience and be capable of coaching the rest of the team.

The role of ops teams (if they still exist) should be primarily in building tools/frameworks for the embedded DevOps engineers to use. From that perspective it's much closer to a traditional internal tools team than a traditional ops team.