News for hackers. Stuff that matters.
Re:Software Developers Should Have Sysadmin Experience by brianwawok on Monday January 09, 2017 @09:15AM (#13356402)
That's terrible. If I'm a Python dev, why should a sysadmin who doesn't even know Python tell me what version of a library I will use?

I have regression tests to catch if an upgrade breaks anyyhing. What does a sysadmin have to approve or deny an upgrade? A little beard stroking and changelog reading?

I think the movement towards containers is like you said, to keep sysadmins off the code. Sysadmins add value in setting up the infastructure and keeping it running. They subtract value when they want to tell developers what version of a library to use.

Re:Software Developers Should Have Sysadmin Experience by bandrami on Monday January 09, 2017 @09:21AM (#13356438)
why should a sysadmin who doesn't even know Python tell me what version of a library I will use

Because he maintains that installation and you don't? But, yeah, that's why virtualenv, Docker, etc. were invented, because devs kept getting sick of installations having consequences.

What does a sysadmin have to approve or deny an upgrade?

Check for conflicts of this version of this library with other software currently in use (by other developers maybe, or even by the same developer). Add it to the watchlist on the dozen or so security mailing lists and newsfeeds he checks daily. Read the changelog and look for implementation problems. Read fora and look for performance problems people are reporting. Yes, beards get stroked during this process, but time and again we see that developers refuse to do this, and wind up coming to us when they break something because of that...

Re:Software Developers Should Have Sysadmin Experience by gnarbarian on Monday January 09, 2017 @10:34AM (#13357019)
Developers tend to fall into the trap of believing we are better than sysadmins but there is immense value in having talented admins around. I have recently experienced this first hand watching someone breeze through server archaeology and vitlrtualization tasks that I struggled through and may never have been able to accomplish in a reasonable amount of time. When admins and programmers recognize each other's strengths and play to them it is a rewarding experience. We just have to realize that we're on the same team.

Also, docker (and containerization in general) is a wonderful thing for both of us. It decouples the fickle apps from systems (also moving targets) and the other apps which are constantly seeking out new and creative version incompatibilities. It makes migration and maintenance a much less frustrating endeavor with fewer surprises along the way.

Re:Software Developers Should Have Sysadmin Experience by btreecat on Monday January 09, 2017 @10:18AM (#13356895)
>Because he maintains that installation and you don't?

So why is that an acceptable mentality for "in-house" developed software but if you buy something proprietary from a third party where you have zero say over what lib/langs are used, it's A-OK?

>Check for conflicts of this version of this library with other software currently in use (by other developers maybe, or even by the same developer).

That's not the case when using containers properly. Every service gets it's own environment so whatever version of lib-xyz is needed, even if incompatible with other parts of the project, are walled off for only the service that needs it.

>Add it to the watchlist on the dozen or so security mailing lists and newsfeeds he checks daily.

Ok this is where I completely agree with you as we have been working on this at our company. My personal solution seems pretty logical though so hear me out.

1) Build a docker file that fully documents the install of your service as well as any OS level dependencies. Ensure that any config files are external to the container to allow sysadmin access.

2) Document in a central location (say an internal wiki) what the external services, servers, repositories, developers, and admins are responsible for the service.

3) Automate builds of containers from repo and add automated testing post containerization.

4) Sysadmins monitor repositories for changes to docker files or wiki articles for new services, databases and libraries as well as taking note of library versions. If an issue with a particular lib or service is discovered, the config files can be edited to point to a new service. Or a new container build can be triggered with zero changes to the source code, but a forced update to the OS packages for the container.

In a tight situation where a developer might not be available on-call, the sysadmins have more control over a similar proprietary product but don't have a workflow for messing with source code (which they are likely not familiar with regardless).

There are solutions to the issues you raise (often trivial ones at that), they just require an adjustment to workflow and an increase in communication between developers and their sysadmins.

Re:Software Developers Should Have Sysadmin Experience by bandrami on Monday January 09, 2017 @10:32AM (#13356998)
I see your point, and like I said I agree this is why Docker was invented and it's the best-in-breed at what it does (namely, being a tourniquet for a self-inflicted wound). My biggest concern really isn't "my problem" since I'm in ops: it's the leftpad worry. I still have teams starting Dockerfiles with "FROM centos:latest" because that's just the mindset they have: "Latest will fix any problems" rather than "Latest will introduce new problems".

And, ultimately, Docker lets that not be my problem, because they have to deal with it when the next leftpad happens. So, yeah: they should have at it. I guess I still think there's something to be said for the cathedral pace, though.

Re:Software Developers Should Have Sysadmin Experience by btreecat on Monday January 09, 2017 @10:59AM (#13357242)
>I still have teams starting Dockerfiles with "FROM centos:latest" because that's just the mindset they have: "Latest will fix any problems" rather than "Latest will introduce new problems".

Well there are two ways to approach this IMO.

One be proactive. Create them a vetted centos or whatever OS environment for them to base off.

The problem is, if you don't keep on top of it as a sysadmin, the developers will just figure out another way to wall you off.

Alternatively, accept that it doesn't matter what underlying OS they use, because a patched OS >> than unpatched and that when done correctly there is minimal exposure even when the service has an exploitable lib due to the jailed nature of containers.

Assuming "latest introduces new problems" too often builds an aversion to patching which can lead to worse issues down the road.

>And, ultimately, Docker lets that not be my problem, because they have to deal with it when the next leftpad happens. So, yeah: they should have at it.

Exactly! The only one responsible for libs are the parties directly leveraging them. Not that developers shouldn't make that info known. It has to be documented to remove the bus-factor of 1, and if it isn't the sysadmins should work with the devs to get it documented.

> I guess I still think there's something to be said for the cathedral pace, though.

I think it depends a lot on your resources as a department/company. You should always execute as quickly as feasible given your team size and work load. Otherwise technical debt has a way of piling up faster than you can offload it.

Re:Software Developers Should Have Sysadmin Experience by closeparen on Monday January 09, 2017 @08:28PM (#13361722)
At my organization, leftpad is not a reason for SRE to tell developers they can't use dependencies. Instead, leftpad is a reason for SRE to run internal package mirrors for all our supported packaging systems (debian, pip, glide, Maven, etc) and ship forks of the build tools so that when you reference a 3rd party dependency, the URL is rewritten to one at our internal mirror. The internal mirror, in turn, goes out and downloads anything it doesn't already have.

They also maintain the base docker images that we're expected to use, as well as the docker build infrastructure.

Facilitation with guardrails, not blockers.

Re:Software Developers Should Have Sysadmin Experience by eightysixfour on Monday January 09, 2017 @11:06AM (#13357307)
I have some perspective as someone who has done a bit of all of these jobs over the last 10 years as well as working in a hosting company that handled release management for large Fortune 500s.

> So why is that an acceptable mentality for "in-house" developed software but if you buy something proprietary from a third party where you have zero say over what lib/langs are used, it's A-OK?

Proprietary software generally has a support agreement and SLA for fixing things instead of getting the response "it works in dev!"

> That's not the case when using containers properly. Every service gets it's own environment so whatever version of lib-xyz is needed, even if incompatible with other parts of the project, are walled off for only the service that needs it.

That's why containers are great, but you have to remember most of the world isn't as fast as this community to adopt things, a lot of things are still being managed the hard way on shared servers with literally thousands of dependencies. Migrating to containers in these instances can't happen fast enough.

> There are solutions to the issues you raise (often trivial ones at that), they just require an adjustment to workflow and an increase in communication between developers and their sysadmins.

Implementing even trivial changes to processes that impact hundreds of people across multiple continents is often not trivial. Devs in India, devs in the US, hosting teams, release management, etc. A lot of those people are doing just enough to get by and not up-to-date tech wise, so not only are you implementing new tools and processes, but you're building out training programs around using them, etc.

These processes are old and will be modernized in time but that's the reality for a lot of "sysadmins."

Re:Software Developers Should Have Sysadmin Experience by dozzie on Monday January 09, 2017 @03:59PM (#13359727)
>> Check for conflicts of this version of this library with other software currently in use (by other developers maybe, or even by the same developer).

> That's not the case when using containers properly. Every service gets it's own environment so whatever version of lib-xyz is needed, even if incompatible with other parts of the project, are walled off for only the service that needs it.

This illustrates why developers should have some experience with administering systems: do not deploy unrelated services on the same machine.

And you know what happens as a byproduct of this rule of hygiene? Suddenly the version conflicts disappear, at least for things that aren't broken anyway.

Re:Software Developers Should Have Sysadmin Experience by e40 on Monday January 09, 2017 @10:08AM (#13356819)
That's the tail wagging the dog (and I manage the sys admin department at my company).
Re:Software Developers Should Have Sysadmin Experience by jasonlotito on Monday January 09, 2017 @02:03PM (#13358813)
> Because he maintains that installation and you don't?

Or not. You're devs are on-call, aren't they? They are maintaining their own software, right?

Re:Software Developers Should Have Sysadmin Experience by xorcist on Monday January 09, 2017 @10:02AM (#13356761)
> why should a sysadmin who doesn't even know Python

Because a Python sysadmin has been through all the transitions of packaging systems, all the nasty corners of "backwards compatible" changes, and knows how underlying changes to the operating system will affect your code, what the storage behaves like under load, and why one tech is not "better" than another. If you really hired an admin (cough, "reliability engineer") for a Python codebase that doesn't know Python, well, that's a different question altogether.

> I have regression tests to catch if an upgrade breaks anyyhing

You don't know what you don't know.

When you can reason about the multiple ways the above statement can fail, congratulations! You are now a seasoned sysadmin, the scorn of junior developers who just want to get things done (who incidentally read a great blog post the other day about a new packaging system that we should immediately transition to and by the way it's all backwards compatible).

Re:Software Developers Should Have Sysadmin Experience by crdoconnor on Monday January 09, 2017 @12:38PM (#13358031)
>You don't know what you don't know.

That's the point of regression tests. The sysadmin also doesn't know. Unless he's the one writing the tests (and IME he's not) or he's painstakingly regression testing everything by hand (trust me, he's not doing that either), making him a gatekeeper for all library upgrades achieves very little except adding bureaucratic friction.

Re:Software Developers Should Have Sysadmin Experience by xorcist on Tuesday January 10, 2017 @06:53AM (#13364198)
Look, do you want a gatekeeper or not? For your small little web project you don't need it, and a little downtime probably isn't catastrophic. But as soon as you are under audit rules you need it, and we call this specialized role the admin. When you grow bigger this will likely branch out to a dedicated change manager, at which point you hopefully have other specialized roles for security as other things as well.

I understand this does not make sense when you are not more people than can fit around a table, but as you grow you will feel the need for more and more specialized roles to fit the changing requirements. The first specialized role is probably the sysadmin (devops, reliability engineer, whatever you call it) and he or she should preferrably be the one on the team with the most knowledge of how things work "under the hood" because that person is the one that can save you when things go haywire. Unless you trust this person to be more knowledgable than you are in those areas, as they rightfully should be, you're going to have a problem.

Re:Software Developers Should Have Sysadmin Experience by crdoconnor on Tuesday January 10, 2017 @07:48AM (#13364436)
>Look, do you want a gatekeeper or not?

No, ideally not - that's the idea behind https://en.wikipedia.org/wiki/Continuous_delivery

Where gatekeepers are required (because regression testing is not yet fully trusted enough for continuous delivery), QA should be the gatekeeper, not sysadmins.

>For your small little web project

My comments are based upon working on projects with a turnover of > ~1-1.5 million USD / day.

>But as soon as you are under audit rules you need it, and we call this specialized role the admin. When you grow bigger this will likely branch out to a dedicated change manager

Every time I've worked with somebody whose role was "change manager" this role was introduced:

* As a response to repeated downtime in the past caused by some kind of idiocy.

* They were required to "sign off" on releases purely as an added bureaucratic step to cover some manager's ass.

* They never once prevented or caught a production issue.

* They always slowed down releases.

>The first specialized role is probably the sysadmin (devops, reliability engineer, whatever you call it) and he or she should preferrably be the one on the team with the most knowledge of how things work "under the hood" because that person is the one that can save you when things go haywire. Unless you trust this person to be more knowledgable than you are in those areas

Ironically the whole idea behind devops (which I fully agree with) is that it should not be a specialized role - developers and ops teams should be blended.

This is precisely because if the two teams are separate and one throws code over the wall to the other then things will go wrong. Then a manager will insist on a gatekeeper.

Re:Software Developers Should Have Sysadmin Experience by warbiscuit on Monday January 09, 2017 @12:15PM (#13357862)
I think a lot of this debate argues for the sysadmin role being part of the dev team. The only real way to get both constraints (production stability and update to date fixes/features) is to have fast feedback between the interest holders of two sides.

In the python-specific case -- the requirements.in / .txt files for the virtualenv should be part of the software VCS, but the sysadmin should be able to edit & pin things just like the devs, so that they can bring their expertise to the container, rather than having to fight it.

---

Mind you, my opinion might not scale - I'm part of a small enough team that I'm holding both those roles, but I try make sure to spent time wearing both "hats", so that one role doesn't get more man-hours clocked.

Re:Software Developers Should Have Sysadmin Experience by crdoconnor on Monday January 09, 2017 @12:42PM (#13358062)
IMO if a sysadmin wants to have visibility into the requirements.txt that's fine.

If they want to enforce a policy of pinning versions, that's very welcome (though I would do that anyway).

If they have specific, relevant comments about upgrades of specific packages - again, fine (though in practice they never do).

If they want to be a gatekeeper for changes to that file they can fuck off.

Re:Software Developers Should Have Sysadmin Experience by jstimpfle on Monday January 09, 2017 @10:10AM (#13356835)
Another perspective: Why don't you want your software to be compatible with the system that your sysadmin provides? (Assuming that system is not completely obsolete).

Minimize your dependencies. It's incidentally also what leads to clean code bases.

Re:Software Developers Should Have Sysadmin Experience by crdoconnor on Monday January 09, 2017 @12:26PM (#13357948)
>Minimize your dependencies. It's incidentally also what leads to clean code bases.

Oh hell no. I have wasted far too much of my life maintaining buggy, technical debt ridden reinvented wheels where there was a well maintained package that could just have been used instead.

Re:Software Developers Should Have Sysadmin Experience by brianwawok on Monday January 09, 2017 @10:54AM (#13357200)
> Minimize your dependencies. It's incidentally also what leads to clean code bases.

You also crush velocity. Smart use of libraries lets you ship code 10x faster. Two identical businesses.. one writes all their own code, one is smart about using libraries. Which one makes it to IPO first, and which goes bankrupt?

Re:Software Developers Should Have Sysadmin Experience by sly010 on Monday January 09, 2017 @12:06PM (#13357797)
Are you arguing that you can keep velocity while building up technical debt?

The company "smartly" using libraries might stuck maintaining a monster of dependencies that only ever was meant to be an MVP. It will require 10x more engineers and while they might move fast at the beginning, they will only slow down over time.

The company minimizing their dependencies and paying attention to their stack will be able to add complexity over time without breaking a sweat. Their costs will be 10th the cost, and they will be able to run profitably.

I am not anti-library or anything, but e.g. adding sci-py to your python project just because you need a gaussian function in one place in your code is just lazy.

Managing dependencies wisely is one of the hardest thing in software development. It's right after cache invalidation and naming things ;)

Re:Software Developers Should Have Sysadmin Experience by nasalgoat on Monday January 09, 2017 @04:45PM (#13360097)
In my experience, the vast number of problems we've had with our mobile apps, especially on Android, has been a developer deciding to use some random SDK to solve a simple problem because he didn't want to take the time to write it himself.

Then the app is broken or has memory crashes, or the final binary is 10x the size it needs to be. 9 out of 10 times it's a third party library. This is why I ban the use of them unless absolutely needed.

Re:Software Developers Should Have Sysadmin Experience by noir_lord on Monday January 09, 2017 @11:42AM (#13357608)
That only holds true (as much as it does) if you assume the goal is always an IPO.
Re:Software Developers Should Have Sysadmin Experience by prodigal_erik on Monday January 09, 2017 @01:07PM (#13358296)
If the world has already written 90% of the code I need, how likely is it that the remaining 10% is valuable enough to make a viable business?
Re:Software Developers Should Have Sysadmin Experience by FanaHOVA on Monday January 09, 2017 @01:49PM (#13358692)
Really likely. Code doesn't mean much, it's mostly about how you present it
Re:Software Developers Should Have Sysadmin Experience by thebeardisred on Monday January 09, 2017 @11:51AM (#13357670)
To call out something that nobody else mentioned: because when you to use that newer version of pycurl or pyASN1 it's probably going to break the system level tools for running patches, handling license enforcement, and being able to keep the database online for other teams.

I've had this problem over and over again (biggest one last being with the US Census). Folks insisted on upgrading a python library and auth to the hosts stopped working.

Re:Software Developers Should Have Sysadmin Experience by user5994461 on Monday January 09, 2017 @03:28PM (#13359495)
> That's terrible. If I'm a Python dev, why should a sysadmin who doesn't even know Python tell me what version of a library I will use?

What if the sysadmin can code in the same language as you, faster and with less bugs?

Re:Software Developers Should Have Sysadmin Experience by dozzie on Monday January 09, 2017 @03:53PM (#13359684)
Better yet: what if the sysadmin can code better and faster in the same language and also three others?
Re:Software Developers Should Have Sysadmin Experience by user5994461 on Monday January 09, 2017 @06:04PM (#13360702)
That would be me!