News for hackers. Stuff that matters.
Re:Software Developers Should Have Sysadmin Experience by AlexB138 on Monday January 09, 2017 @09:51AM (#13356672)
As a Sys Admin turned Automation/Tools Engineer, I think you're missing part of the point. You've got the beginning of it right in saying that Sys Admins used to be involved in pinning down versions, and even in why that was necessary, but I believe you're incorrect in saying that the containerization technologies are bad for removing that.

Those technologies don't exist so Developers can get around Sys Admins and ignore your helpful advice. They exist to solve that problem that makes the Sys Admins role there necessary. It removes the underlying need for a Sys Admin to worry about the versions. Admins should see this as a good thing, but in my experience many dislike it because it takes them out of their Gatekeeper role. We shouldn't WANT to stop the Developers from from having the newest version of things. They aren't kids playing with toys that we need to nanny over, they're doing work that creates values and the fewer things we do to get in the way of that, the better.

If something breaks due to version changes, their testing should catch it. If things are breaking in production, we ought to get involved because there's some other problem, but before that we, as a profession, need to learn to get out of the way and let people work by letting technology handle the problems. The "Gatekeeper" mentality needs to die as quickly as it possibly can.

Re:Software Developers Should Have Sysadmin Experience by jstimpfle on Monday January 09, 2017 @10:05AM (#13356788)
> They aren't kids playing with toys that we need to nanny over

In my experience (university), yes they are, and they should do that at home.

Why do you need the latest bleeding versions in the first place?

In my sysadmin experience, people believe software gets bad and deprecated as soon as the glory next breaking version appears. I don't think I need to argue why this is an illogical stance.

With my developers hat on, bumping to the next version mid-process reliably introduces more friction than is worth it. People think the next version solves that one weird issue but ignore that it introduces two new ones and that the software must be changed to fix five new incompatibilities.

But the solution reliably is to just not use that weird feature that caused the bug in the first place, and think what a clean solution would have been. And guess what, the result is a cleaner and more compatible code base. It's a tip that works for me again and again: If there is friction, think - before spending the next hours with an update that will soon lead to new problems.

It's great that you can for example compile Linux without too much friction. It's great that arcane shell scripts can run on any system. Stability (in a compatibility sense) is not a nice-to-have, it's basic sanity.

Re:Software Developers Should Have Sysadmin Experience by treehau5 on Monday January 09, 2017 @10:59AM (#13357238)
My comment to you two is -- why not both?

Stability, sanity, all that is amazing, and a must have.

But also bug-fixes, security improvements, and performance improvements are wonderful too, which tends to come with using up-to-date dependencies.

The problem with the latter, as you mentioned, is when it introduces breaking API changes and is wholly not backwards compatible. This is not a "kids playing with toys wanting to experiment problem" this is a bad software problem, which is why I like Go, and why I liked Java when I was doing it full time. If the language you use has backwards compatibility as a first-class citizen, most likely the package authors will act that way too, and then the maintainers, and eventually the developers. Limit your software choices to those who care about not breaking everyone's shit every 2 weeks. Heck even when I write my own API's now that I know only my company is going to use internally I am thinking about this.

Re:Software Developers Should Have Sysadmin Experience by brightball on Monday January 09, 2017 @11:30AM (#13357519)
Backwards compatibility is seriously under appreciated. When I tell developers to ensure that their changes are backwards compatible, they tend to look at me like I'm green.

I do not understand the disconnect that developers have with understanding all of the benefits that it brings. Yes, you have some extra code in your code base so it's less clean. You also have a stable environment as a result. The first affects only your personal preference. The latter affects all of your developers and users.

Unless you have a situation where it's impossible to maintain, not insisting on it is pure self interest.

Re:Software Developers Should Have Sysadmin Experience by gaius on Monday January 09, 2017 @11:47AM (#13357639)
I do not understand the disconnect that developers have with understanding all of the benefits that it brings.

Because they've never worked on an old codebase, because front-end technologies change so often and everything just gets re-written anyway. It's a waste of time worrying about this when the code won't make it to is first birthday.

If you were speaking to seasoned C and DB developers about stability in the tools and the platform, you'd be preaching to the choir.

Re:Software Developers Should Have Sysadmin Experience by bandrami on Monday January 09, 2017 @12:25PM (#13357939)
This gets to the complaint that so much of the open source ecosystem gets to version 0.8.6 (whether it's named that or not) and then completely rewritten "this time the right way". That's not actually a good thing.
Re:Software Developers Should Have Sysadmin Experience by prodigal_erik on Monday January 09, 2017 @12:58PM (#13358205)
As jwz put it,

> It hardly seems worth even having a bug system if the frequency of from-scratch rewrites always outstrips the pace of bug fixing. Why not be honest and resign yourself to the fact that version 0.8 is followed by version 0.8, which is then followed by version 0.8?

Re:Software Developers Should Have Sysadmin Experience by sgift on Monday January 09, 2017 @11:52AM (#13357693)
> Yes, you have some extra code in your code base so it's less clean. You also have a stable environment as a result. The first affects only your personal preference. The latter affects all of your developers and users.

Backward compatibility has real costs. You cannot restructure your code base as easy, you cannot deprecate bad ideas, cannot extend it as easy more and so on and so on. Sure, it also has real benefits (as you've stated), but missing the disadvantages while only highlighting the advantages is not a useful approaching, it only shows "your personal preference".

Re:Software Developers Should Have Sysadmin Experience by serge2k on Monday January 09, 2017 @02:31PM (#13359052)
> Yes, you have some extra code in your code base so it's less clean. You also have a stable environment as a result. The first affects only your personal preference. The latter affects all of your developers and users.

No, the former results in a bloated code base full of old legacy crap that no one understands and is afraid to touch because it might break. You have to insert weird workarounds because that bug is now a feature to some idiot and you provide backwards compatibility so it lives forever now.

Re:Software Developers Should Have Sysadmin Experience by StillBored on Monday January 09, 2017 @02:57PM (#13359274)
Has software quality increased now that everyone is refactoring and rewriting everything for every release?

I don't think so.

Its just plain arrogance to believe your are a better developer than the guy who came before. Having some fear of breaking the code base is healthy, the same way that having a little fear that the chainsaw is going to cut off your leg makes you safer.

Re:Software Developers Should Have Sysadmin Experience by jstimpfle on Monday January 09, 2017 @11:13AM (#13357372)
I like your answer. It's civilized, balanced, and I agree with every word of it :-)

I'll add as an anecdote that I do follow your practices (limiting dependencies). It works wonderfully on Debian stable (most of the software there is now >2 years old, the next version has just been soft frozen). I have the occasional package pulled from testing: I recently toyed with Perl6. And currently I use a newer version of python3-sphinx for a nicer doc syntax but I could do without. It causes no headaches at all.

Re:Software Developers Should Have Sysadmin Experience by Declanomous on Monday January 09, 2017 @03:03PM (#13359337)
For one thing, if there is any place to treat software like a toy and to play around with the latest version, it would be a college or university.

I don't particularly care about having my software on the latest version. I personally prefer using the old version for six months while the newest version gets the bugs worked out of it.

I know sysadmins value reliability and security, but it's really frustrating when every upgrade takes dozens of hours of work to approve. Questions like "What features do you need in the new version" miss the point. It isn't about the features of the software, it is about maintaining a modern code base.

Upgrades always have the potential to break things, but when you keep up with the upgrades it is easier to achieve the stability and security goals the sysadmin wants. When you upgrade often, it is easier to read the documentation and find where changes might break something, and when things do break it is easier to fix them. Upgrades that jump over several versions at a time are a nightmare to debug, and it creates a lot of technological debt that you have to work out later.

Ultimately, sticking with a version of software because it works is trading a little stability now for an absolute mess down the line.

Re:Software Developers Should Have Sysadmin Experience by jstimpfle on Monday January 09, 2017 @06:37PM (#13361005)
I said it elsewhere, but personally I'm a Debian stable evangelist. There is one major upgrade every 2 years or so. It often goes without friction. The rest is mostly security updates. Breakage between major upgrades is very rare.

I don't think this thread is about "maintaining a modern code base" at all. Whatever that should mean -- My impression is you've fallen victim to the hype train.

In my perception the thread is about always catching up with the latest and greatest. Would you say in all earnest that my code is not modern because I make a point of developping against solid standards and not constantly longing for things that are not in my distribution (the software there is usually 0.5-2.5 years old)?

You can check some of my code at https://github.com/jstimpfle. Is it "not modern"? I'm a reasonable but not outstanding developer, and not saying that everything will work on your computer (since I'm usually the only tester) -- but I'm pretty sure I can get everything there to run on your computer with minor effort.

> When you upgrade often, it is easier to read the documentation and find where changes might break something, and when things do break it is easier to fix them.

No. Breakages are less frequent because the software is not brand new, and they are better known because all people using the stable release are on the same version. Documentation comes with the distribution, but I don't have any problems googling it by giving the version string either.

> Upgrades always have the potential to break things, but when you keep up with the upgrades it is easier to achieve the stability and security goals the sysadmin wants.

This thread was never about security and I don't approve. I don't think you are familiar with the concept of a stable release.

> Upgrades that jump over several versions at a time are a nightmare to debug, and it creates a lot of technological debt that you have to work out later.

No. If you develop against solid standards you have less breakage. It's not about incompatibility with the most recent versions. That would be a stupid idea. It's about compatibility with releases other than the latest and greatest. This means not depending on the hot new features that are only in these versions, simple as that.

Re:Software Developers Should Have Sysadmin Experience by flukus on Monday January 09, 2017 @07:21PM (#13361339)
> I said it elsewhere, but personally I'm a Debian stable evangelist. There is one major upgrade every 2 years or so. It often goes without friction. The rest is mostly security updates. Breakage between major upgrades is very rare.

That's fine for an OS, but what do you think business customers would say if you said "sorry, that feature won't be added until the next release in 2 years time"?. That's were tools like pip come in, they let the software move faster, which it often needs too.

Re:Software Developers Should Have Sysadmin Experience by bandrami on Tuesday January 10, 2017 @09:15AM (#13364840)
what do you think business customers would say if you said "sorry, that feature won't be added until the next release in 2 years time"?

We say that all the time; we have a two-year release cycle. And in our field (aviation) that's considered breakneck.

Re:Software Developers Should Have Sysadmin Experience by jstimpfle on Monday January 09, 2017 @07:25PM (#13361368)
So what exactly are the latest-and-greatest libraries that you absolutely need to implement your own business critical features?

Please list more than only one. It's simple to make exceptions for exceptional requirements.

Re:Software Developers Should Have Sysadmin Experience by flukus on Monday January 09, 2017 @07:35PM (#13361431)
Well for me, all of the libraries I use because none of them exist as packages for any OS.

With most of them security fixes will only go into the latest version though, so once you get behind your system is insecure.

Applications aren't something you build and forget. An unmaintained project is a dead one.

Re:Software Developers Should Have Sysadmin Experience by jstimpfle on Monday January 09, 2017 @07:56PM (#13361574)
> Well for me, all of the libraries I use because none of them exist as packages for any OS.

In conclusion you don't use any libraries that are packaged for any OS.

What libraries?

Also assuming that some libraries you use don't exist for your OS, that doesn't mean that you absolutely need the latest and greatest in a business critical way. So, not approved.

All in all, not too fond of the reasoning and the evidence you provide.

Re:Software Developers Should Have Sysadmin Experience by flukus on Monday January 09, 2017 @08:07PM (#13361624)
Your logic would prevent any app from just about any non-c ecosystem running. Java, ruby, python, dotnet, rust, go, they all have their own library management and very few of those libraries will be available in an apt repository (let alone a compatible one).

Your policy may work in a university, but you'd be fired from any real business.

Re:Software Developers Should Have Sysadmin Experience by jstimpfle on Monday January 09, 2017 @08:38PM (#13361779)
You are still not providing evidence.

You're also making bold claims ("any app from just about any...") that my personal experience can just not validate. It's very easy to write applications without fancy dependencies. Recently I did algorithms, systems and applications in C, C++, python, sdl, alsa, X11, Unix shell scripting, lp_solve, and some web programming in python, javascript, sqlite3. All rock solid and stable - all will probably run on any Linux box from the last 5-10 years (python3 came only in about 2008; forget about the lp_solve bindings and just use the command-line tool).

There are 2050 python3-* packages on my system. Not that I think it's a good idea to use most of them. What's "compatible"?

So what are the libraries you absolutely need? What is this week's secret sauce?

Re:Software Developers Should Have Sysadmin Experience by flukus on Monday January 09, 2017 @09:26PM (#13362031)
.net MVC, nUnit 3, jquery, knockoutjs, and nHibernate to name a few. I could name a dozen similar tools on the java stack. Pretty soon I'll be experimenting with rust with libui.

>There are 2050 python3-* packages on my system. Not that I think it's a good idea to use most of them. What's "compatible"?

2000 of them are random versions someone made a package for that are unknown to the core team and probably not receiving updates.

Your list of projects sounds like typical academic ones, not tools used by businesses that employ most software developers.

You're also missing the other benefit of these tools, that we develop against the deployed version. There are no compatibility issues because we develop on ubuntu and host production on redhat.

Re:Software Developers Should Have Sysadmin Experience by jstimpfle on Monday January 09, 2017 @10:21PM (#13362289)
If you consider making websites, and tools and web applications for internal processes, academic... I also did some contract work where I created a server and a client for displaying advertisement media to commodity screens, in a tiled fashion. The tools were all there, C, python, bash, X11, some media libraries, the versions there were all fine (I worked around a bug in mplayer though).

jquery, knockout. I don't know a first thing about bundling dependencies for client-side javascript code (not going super fancy there, don't need jquery, knockout - tried writing single page apps by hand but they are hugely complex) but anyway, isn't that independent from server installations? Don't you bundle these libraries in-tree? If so, it doesn't relate to the discussion.

.NET... It's MS, do you run on Mono? How does the question of requiring the latest version apply?

Re:Software Developers Should Have Sysadmin Experience by mdekkers on Tuesday January 10, 2017 @10:31AM (#13365501)
> There are no compatibility issues because we develop on ubuntu and host production on redhat.

Sorry, but that is a hilarious argument, almost straight from Gentoo is Rice: https://fun.irq.dk/funroll-loops.org/

Re:Software Developers Should Have Sysadmin Experience by mdekkers on Tuesday January 10, 2017 @10:18AM (#13365377)
> Your policy may work in a university, but you'd be fired from any real business.

Nah. If we define "real business" as something with a decent turnover, employing over 250 people, and being in business for over 8 to 10 years; a business that isn't actually in the business of writing software (the majority of what makes up global stockmarkets, or "real business" in most peoples' eyes) then you will find that OP's attitude and policy-making philosophy is right on the money. (Source: Was CTO in exactly the above type businesses for many years)

Re:Software Developers Should Have Sysadmin Experience by jtreminio on Monday January 09, 2017 @11:20AM (#13357436)
> Why do you need the latest bleeding versions in the first place?

Because the newest version has several features that we would like to take advantage of immediately?

Look at PHP 7.0 which introduced return types, and 7.1 which introduced nullable return types. These are features I really want in my application, so we upgrade.

Re:Software Developers Should Have Sysadmin Experience by rch on Monday January 09, 2017 @11:08AM (#13357327)
No offense intended, but your university probably isn't competing for top engineers. Grad students and postdocs aren't professionals yet either.

The sysadmin role has traditionally been a focus in that environment (e.g. controlling access to cluster resources).

Re:Software Developers Should Have Sysadmin Experience by jstimpfle on Monday January 09, 2017 @11:15AM (#13357395)
Define "professional". And let me claim that "top engineers" are actually the prudent ones - which you didn't refute.
Re:Software Developers Should Have Sysadmin Experience by rch on Monday January 09, 2017 @01:17PM (#13358385)
I think we agree on the prudence of professional engineers.

The definition of 'professional' is up for debate, but I'd encourage people to weigh in on the following (to IEEE, not me):

- an appropriate engineering education background (ABET/EAC)[1]

- at least four years of engineering experience in your field and under the supervision of qualified engineers

- passed two exams (the Fundamentals of Engineering [FE] exam, which is now a computer-based test available essentially year round, and the eight-hour PE exam)

- kept current by as a minimum meeting your state's continuing education requirements.

-- http://insight.ieeeusa.org/insight/content/careers/97473

[1] I think it would be worthwhile to consider apprenticeships, equivalent to the 'law office study' path to attorneys' bar certification.

Re:Software Developers Should Have Sysadmin Experience by shuntress on Monday January 09, 2017 @11:26AM (#13357483)
I think the pain of keeping any project fully up-to-date is (sometimes far) less than the pain of updating an out-of-date project.
Re:Software Developers Should Have Sysadmin Experience by jstimpfle on Monday January 09, 2017 @11:40AM (#13357591)
If it is, your project has a problem.
Re:Software Developers Should Have Sysadmin Experience by mancerayder on Monday January 09, 2017 @01:09PM (#13358327)
Those technologies don't exist so Developers can get around Sys Admins and ignore your helpful advice. They exist to solve that problem that makes the Sys Admins role there necessary. It removes the underlying need for a Sys Admin to worry about the versions. Admins should see this as a good thing, but in my experience many dislike it because it takes them out of their Gatekeeper role. We shouldn't WANT to stop the Developers from from having the newest version of things. They aren't kids playing with toys that we need to nanny over, they're doing work that creates values and the fewer things we do to get in the way of that, the better. If something breaks due to version changes, their testing should catch it. If things are breaking in production, we ought to get involved because there's some other problem, but before that we, as a profession, need to learn to get out of the way and let people work by letting technology handle the problems. The "Gatekeeper" mentality needs to die as quickly as it possibly can

Well - that's all well and good, when put such that "Gatekeepers" are viewed as blockers.

However, we "Gatekeepers" are the ones that get paged and / or yelled at by a CTO when an application keels over. Not the developers. The developers get to sit in their sandbox (otherwise known as "production" in 2016/17) of ever-changing library versions that were only rapidly tested in QA. Then they play a game of Starcraft II, scan HN and go to bed. When something runs out of memory or crashes in the middle of the night, we get paged. So, hell yes we should be involved in the process.

Sincerely,

Gatekeeper

Re:Software Developers Should Have Sysadmin Experience by snuxoll on Monday January 09, 2017 @02:09PM (#13358866)
Sounds like an organizational failure to me.

When I started at my current company the traditional silo between dev and systems was there (although we were allowed to deploy our own stuff) - they managed everything we ran our apps on and we just deployed them to servers they had already configured. Over the past ~3 years we've made a lot of changes, the department manager for our IS team is present in our daily standup calls to relay information between our two teams and we now have a couple separate VMWare clusters dedicated to our applications and VM running on them is our responsibility for the most part. We are the first to get called for issues with our applications, and where necessary we work collaboratively with our systems team to resolve them - we don't throw blame around, it does no good.

I should add most of this is only possible because we have real DevOps people on our team (well, really, it's just me right now - we lost our other and need to hire a replacement still) - not developers who know enough to copy a blob of crap to a server to run, but people who have real skills in both aspects. We are trusted to maintain things because we can do it right, and while it took a lot of work (and some unfortunate infighting) to get to this point both of our departments are working great with this arrangement.

There's still kinks that need ironing, we've not done an adequate job at writing documentation so our systems team can help with some failures (primarily on our Linux VM's, our whole systems team is Windows admins) if we aren't available - but it's on the radar as well as getting PagerDuty set up to escalate alerts to them if we don't respond in time (like having our PostgreSQL data volume fill up over the weekend, not a call I wanted to get at 10AM on Sunday).

So yeah, fix your culture issue, get people communicating daily between your teams, share responsibility for issues instead of placing blame.

Re:Software Developers Should Have Sysadmin Experience by jasonlotito on Monday January 09, 2017 @01:59PM (#13358767)
> However, we "Gatekeepers" are the ones that get paged and / or yelled at by a CTO when an application keels over. Not the developers.

And that's why people are moving away from that model. It's part of the reason DevOps is being embraced as a model. Developers should be on call to support the applications they build. You get benefits all around.

Re:Software Developers Should Have Sysadmin Experience by CaptSpify on Monday January 09, 2017 @05:13PM (#13360297)
should is the key-word here. As a sysadmin, I'd love to work closer with devs, especially during outages. Unfortunately every time we bring up on-call, the room goes silent. This is very anecdotal, but IME, devops has just become a way for devs to bypass sysadmins.

I wonder how many companies are doing it right vs doing it wrong? Any anecdotes from a proper devops group?

Re:Software Developers Should Have Sysadmin Experience by serge2k on Monday January 09, 2017 @02:24PM (#13358982)
and we get yelled at when we can't deliver a feature because some gatekeeper is sitting atop his little throne in the kingdom of servers saying no. ;)

This is institutional failure.

Re:Software Developers Should Have Sysadmin Experience by CaptSpify on Monday January 09, 2017 @05:16PM (#13360313)
Then place the blame on the gatekeeper. As a sysadmin, I'd be more than happy with you pointing the finger at me as the reason why you can't deliver a feature. Assuming, of course, that you've run the proper tests and gotten QA's approval.
Re:Software Developers Should Have Sysadmin Experience by user5994461 on Monday January 09, 2017 @03:24PM (#13359475)
Honestly. Developers should stop thinking that their jobs is to release features all the time at all costs. That's simply not true and that's counter productive for the business.
Re:Software Developers Should Have Sysadmin Experience by WorldMaker on Monday January 09, 2017 @03:37PM (#13359566)
If only the business might learn that it's not their jobs to request new features all the time at all costs...

If only customers weren't fickle and might learn not to demand new features all the time whatever the costs...

It's turtles all the way down.

Re:Software Developers Should Have Sysadmin Experience by user5994461 on Monday January 09, 2017 @05:53PM (#13360621)
And in the end, that's the developer who always gets the decision. Does he ships half assed half finished every single time, or does he take time to do some testing and not break production.
Re:Software Developers Should Have Sysadmin Experience by serge2k on Monday January 09, 2017 @03:44PM (#13359615)
Developers do what the business wants them to.
Re:Software Developers Should Have Sysadmin Experience by jstimpfle on Monday January 09, 2017 @02:29PM (#13359035)
Example?
Re:Software Developers Should Have Sysadmin Experience by jdbernard on Monday January 09, 2017 @02:05PM (#13358829)
The problem with your characterization of things is companies like the one I'm in where us "Developers" have replaced the gatekeepers entirely. We have no dedicated SysOps, and yet our production environment stays up just fine.
Re:Software Developers Should Have Sysadmin Experience by zeveb on Monday January 09, 2017 @12:34PM (#13358005)
> It removes the underlying need for a Sys Admin to worry about the versions.

No, they really don't: they remove the ability of system administrators to administer versions of software across the total system.

This is bad, e.g. when a new OpenSSL vulnerability comes out (it being a day ending in -y) and every piece of software has to be updated.

> We shouldn't WANT to stop the Developers from from having the newest version of things. They aren't kids playing with toys that we need to nanny over, they're doing work that creates values and the fewer things we do to get in the way of that, the better.

I am a developer, and I disagree. We are, by and large, kids playing rather than adults making carefully considered decisions. We'd rather use v3.0.rc-1-awesome rather than 2.17.12, because the former is the version that adds an API that saves us from writing twenty lines of code, never mind that it also is untested, unstable and very likely insecure.

We need adult supervision. We need oversight. That's why I argue for using stable, LTS-style distributions, and running against the distro packages unless there is a very good business reason not to (and yes, 'we can't implement necessary functionality in a cost-effective timeframe' is a valid business reason). I'm not opposed to using the bleeding edge when it makes business sense; I'm opposed to developers using the bleeding edge because they like it, and keeping the business in the dark.

Re:Software Developers Should Have Sysadmin Experience by eternalban on Monday January 09, 2017 @11:28AM (#13357499)
I agree with OP and would amend his statement to "15 years ago, when a project would kick-off, [sysadmins & architects would] be invited ..."

From an architectural point of view, microservices take the reductionist approach to system design to an absurd limit, and per my professional experience (fwiw & ymmv) are due to the general architectural illiteracy of the rank and file practitioners in this field.

Re:Software Developers Should Have Sysadmin Experience by gaius on Monday January 09, 2017 @11:50AM (#13357664)
Right, microservices isn't "architecture". It is - whether they know it or not - an admission that "we can't do architecture so we'll chuck it over the fence and let the ops people worry about how it all hangs together".
Re:Software Developers Should Have Sysadmin Experience by eternalban on Monday January 09, 2017 @12:00PM (#13357752)
> microservices isn't "architecture".

Yes the 'no-architecture architecture'. It's very Zen. /s

> how it all hangs together

In case you are interested in rescuing :) young but promising talent in the field, next time you find yourself involved in a discussion about microservices "architecture", point out the realization of a single-node application per this approach, where every function is a process, and the call stack requires IPC, and the 'linker' is considered obsolete and outdated technology.

Re:Software Developers Should Have Sysadmin Experience by gaius on Monday January 09, 2017 @12:03PM (#13357774)
I once worked on an application that - no joke - comprised 5000 VMs each of which was running one "service" in a dedicated JBoss. It was laughably bad.
Re:Software Developers Should Have Sysadmin Experience by iask on Monday January 09, 2017 @10:31AM (#13356992)
At what point do you determine who is responsible for securing the environment? The "Gatekeeper" mentality is stemming from this. There is no clear line in any organization and I see the blame-game all the time.
Re:Software Developers Should Have Sysadmin Experience by whyileft on Monday January 09, 2017 @10:46AM (#13357123)
This is exactly the example I give when I try to explain to someone that managing through personal responsibility as opposed to team and organizational responsibility will grind your company productivity to a crawl.
Re:Software Developers Should Have Sysadmin Experience by amorphid on Monday January 09, 2017 @10:38AM (#13357056)
That's kind of like asking which employees at a bank are responsible for keeping cash in the vault. Hopefully it’s a group effort.
Re:Software Developers Should Have Sysadmin Experience by astine on Monday January 09, 2017 @11:09AM (#13357338)
There are actually generally rules for which employees have responsibility over the vaults in banks. The employees aren't equally responsible. Roles and responsibility are defined strictly. Generally, the lowly tellers aren't allowed the same access to the money as the general manager, and they aren't held responsible to the same degree if money in the vault goes missing.
Re:Software Developers Should Have Sysadmin Experience by KallDrexx on Monday January 09, 2017 @11:59AM (#13357745)
This has actually been a very insightful response to me.

After reading your comment it now occurs to me that Docker and other container systems are actually a huge organizational tool. One issue I have encountered at companies is keeping the IT and development departments on the same high level organizational incentives to keep political barriers from coming up between them (and conflicts arising).

Containers can help keep everyone's incentives aligned because System admins can focus on the actual administration aspects of the systems and infrastructure (that devs do not need to be concerned about, like vnet layouts and whatnot) while devs can focus on the actual development and deployment without having to have everything confirmed and approved by the IT departments.

Re:Software Developers Should Have Sysadmin Experience by eropple on Monday January 09, 2017 @12:24PM (#13357927)
At almost every shop I've ever seen, "sysadmins" are also the ones whose responsibility it is to at least attempt some sort of security practice and business continuation. Which opaque, "just run it" containers actively fight against. Did the developers actually audit what they've pulled in as dependencies? Did they make sure that they can be rebuilt if whatever package source goes away? Where is everything documented? Containers, as usually implemented, rather than "keep everyone's incentives aligned", instead damages the ability of the adults in the room to keep everything from falling apart.

(I have been both the sysadmin saying "no" and the developer mad at sysadmins saying "no". But going slower and doing our homework has never, ever hurt me or my employers.)

Re:Software Developers Should Have Sysadmin Experience by KallDrexx on Monday January 09, 2017 @06:21PM (#13360864)
I've never seen sys-admins verify security of dependencies outside of the major dependencies (like the language VM version). Security wise they have been much more concerned with the security of data storage systems (databases, elastic search, etc...), the operating system, and the network in general.

Security of the application is very much so responsibility of developers, not system admins, as the developers have the best point of view to understand the implications of the software they are developing/integrating with.

If there are routine violations of security at the application level that aren't being caught by developers working with those systems then the company as a whole needs to sit down and make sure the development teams have the proper security procedures in place, because putting a department in charge of security that has all accountability but no power to remedy the situation is a recipe for political fights between departments and a disaster. Proper code reviews and team leads with experience should be able to catch more security issues than sysadmins will.

If your sysadmins are in charge of security review of the application then they have to be in charge of security review of every low level dependency at the individual package level. Otherwise your developers won't think about it because it's not their problem (IT will review and let me know if anythings bad) and it encourages them to lack accountability of the security of their own software.

Re:Software Developers Should Have Sysadmin Experience by SomeStupidPoint on Monday January 09, 2017 @10:06AM (#13356802)
Doesn't that constant updating and lack of version oversight create security risks?

Developers may not be as aware of those topics as sysadmins.

Re:Software Developers Should Have Sysadmin Experience by ajford on Monday January 09, 2017 @10:44AM (#13357108)
Isn't it just as bad to keep versions locked for months and years so we don't upset the delicate balance of versions on the production env?

I've worked at a place where they were running PHP 4.4.9 until about 8 months ago. And they were upgrading to 5.4! I get that it was work to convert a lot of the older code base to 5.4, but it was already passed EOL when they were switching to it. And 5.5 wasn't much behind it.

So now in the near future, they'll need to upgrade again (though they probably wont), and they'll probably jump to 5.6, which EOLs in two years (probably two years after it EOLs).

Re:Software Developers Should Have Sysadmin Experience by UK-AL on Monday January 09, 2017 @03:56PM (#13359703)
The lack of updating is also a massive security risk.