As a sysadmin, I'm fine with the concept of continual improvement, there is nothing worse than a static industry. The job is changing every day and this.is.good.
What I understand from this post is that startup people don't want to deal with sysadmin anymore. It makes a lot of sense, considering the amount of time/energy/money you need to run a stable infrastructure. But at the same time, heroku, ec2 & al. will never be able to cover everybody's needs. So while I'm convinced that a dev or devops will be fine deploying his webapp to some random cloud, I'm also convinced they will come crying when massive load hits them and they can't tune the system.
The sysadmin is not only the guy who configure new servers, it's also the guy who explains the new dev dude that, no, he can't do 15 inner joins on large tables to create the home page because it will stress the IOs so bad that the whole website will collapse.
Ease of use != flexibility.
About ten years ago I remember talking to some old timers who were surprised that a lot of the younger sysadmins no longer even did any perl progamming, but rather did things like program Cisco rounters and set up Oracle servers.
At Google we don't have sysadmins at all, but rather "Site Reliability Engineers", and they are expected to be able to answer some fairly deep C/C++ data structure and algorithms questions. All of those skills are necessary to run a Google-sized infrastructure, and it's quite a far cry from the sysadmins that don't even write perl scripts any more -- and yet feel perfectly free to call themselves "sysadmins".
When I first started, I both wore a pager and wrote C programs --- including the 'ninit' program that was described in the Unix haters handbook. The fact that I was one of the people who might be woken up at 3am when the mail server melted down due to named core dumping informed how I wrote my C programs (i.e., very defensively), and is a prime example of the dev/ops concept.
But why are so many devs shooting at the sysadmins these days ? I never read anything from sysadmins complaining about the astonishingly low level of knowledge of a lot of devs. And they are numerous, willingly ignoring even the most basic performance issues and vomiting thousands of lines of codes, hundreds of tons of libraries, to simple problems.
It seems that devs are frustrated to not be able to push their random app to the web in 5 minutes, and throw their love to PaaS like an apple geek discovering the new iphone, blaming the sysadmins for not providing the same thing at their own company without even understanding (or trying to understand) how operating systems work, and how their own code should be built.
The real issue here is that when devs and sysadmins DON'T work together, and ignore each others problems, constantly blaming each other, then the result sucks. When the sysadmins are 20% devs, and the devs 20% sysadmins, and they drink their coffees together in the morning, you don't see that kind of problem. They built good, reliable, scalable infrastructure, whether it's in the cloud or in the basement of their house.
They should be. At my current employer we're pretty good about this, but in previous misadventures I've run into a lot of people who don't even understand HTTP or how a web server works (if I have to explain thread-per-connection to a guy who's been doing web development for five years, there's a problem!).
Remember, your developer spends their time making it work on a dev box, or even their own laptop. Performance testing and such is an art that if they had time to learn it... well I guess they'd be a sysadmin.
If you actually give a shit about web development beyond a paycheck, this is stuff you just gotta know. If you don't understand the system, your assumptions and assertions are so much more likely to be untenable that it becomes dangerous.
The below questions wouldn't fly at my current employer due to a lot of legacy concerns, but my own list of interview questions would include a number of stack questions in addition to programming questions:
-Explain asynchronous I/O to me. -How do you have to structure your code to leverage asynchronous I/O? (Alternately phrased: what happens if you write code that depends on synchronous I/O operations within an asynchronous I/O web framework?) -You've got 9 memcached servers with a cache hit rate of 80% resulting in a database load of 80%. What happens if you naively add another memcached server to your cluster? (a question our VP of engineering actually does ask, and I stole it 'cause it's great)
You don't have to be a hardcore nginx source code diver to understand your stack. You just have to understand the principles behind it well enough to make good decisions. A sysadmin's job isn't to audit your code for your terrible understanding of the model on which your stack is predicated.
First, I don't think it's good for a developer to have all the crap in their head that I do in mine. They'd be spending way more time pre-optmizing code than they would just getting stuff done.
A good developer will write a well thought out TDO that infrastructure and business units can provide feedback to.
Your questions about asynchronous i/o and memcached are great, and if memcached was a requirement for a sysadmin at a position it would be a good question to make sure they understand the software.
As a sysadmin though, you're also balancing
- Can the new system exist on an existing server/chroot/virtual machine - What monitoring is necessary for the application afterwards - Does the application meet all the standard requirements for database access ... out PFC is over 100 steps, this is just some of the bigger ones
- Can the sysadmin support the system after go live? The developers job is to create it, the sysadmins job is to keep it running long term.
Also, you are correct. The sysadmin shouldn't be auditing your code. The sysadmin should have a stack in place that they know works, and be able to help you do one of 2 things:
- Give advice on how your system can best fit in that stack - Help you change the stack to best fit your application
Which direction you go on those two choices depends on the nature of the application, how mission critical is, required timelines for launch, budget and more.
I don't say this stuff just for show--I've been there and done that, and it's stuff I've done to become what I consider an acceptable developer. I can, and have, set up a three-tier fully-enterprise-up-the-ass application infrastructure on my own (I have a couple machines on a VLAN at home as a VM farm) in order to learn this stuff. A developer who can't work the stack is a developer I don't want touching my code, because they're a developer that doesn't understand what they're doing actually does. Abstractions are great, but every abstraction leaks--they should be treated as productivity enhancers, not as insulation from knowing what you're doing.
Nobody can know everything about everything (there's a guy here at my current job who has forgotten more about the JVM than I've ever learned), but you gotta at least be able to talk about it. If a developer can't explain to me high-level concepts like how your web server handles threading, I don't even know what to say. This stuff is just so integral it's not even funny. You just gotta know it. Specialization is for insects.
.
EDIT: I think you and I are coming at this with very different philosophies. At my current employer, we don't have a PFC. We don't have TDOs. We have two live-site guys and a few operations engineers who are able to jump into our codebase (one is probably the most knowledgeable developers at the company who wanted a change). We have developers who can be trusted to make good decisions, and we have senior devs and managers who are very well equipped to analyze the plans and ideas that come from junior devs to say "hey, that won't work." But to be able to trust developers, they've got to understand how stuff works. (We don't ensure full-stack knowledge as well as I'd personally like, but we do a pretty good job of it--at the least, there are no "front-end engineers" who can't sling Java and no Java guys who can't write HTML/CSS/JS if a project comes along that requires it.)
The deck is stacked against sysadmins - developers are the people who deliver business value to leaders, and sysadmins are the ones who cost money and say 'no'. When you are on a tight timetable, the person who says "sure" is the one who gets the high fives... assuming things work out, of course.
This happens when the company discovers that when the system goes down they stop making money. If you've managed to design and build a system to be very resilient and reliable on the chosen platform and has proved to need minimal maintenance, then chances are good you have a very small ops team who doesn't get in the way much.
There are plenty of DBAs complaining about the low knowledge level of a lot of devs, e.g. using an ORM instead of writing SQL and wondering why the app runs like a dog.
I would say, 25 years later, the need to fix C bugs is far less relevant. That said, I'd say around 50% of the system administrators I've worked since moving into the Unix world know C and can at least read the source code of the operating system they're supporting. Even if there's not much need to write anything or even submit bugfixes anymore. Also, the number of unix admins I know who don't know a scripting language well enough to write a reasonably-sized, useful tool is vanishingly small, and those few have been very solid with some other technology(network, storage, etc.) and valuable anyway.
"System Administrator" is a broad term, just like "Developer." I'm not sure google has either, right? If we're going by fancy titles, Google has "Software Engineers." I'm not going to criticize Google's culture but the fact is if I say "Software Developer" you know what I'm talking about and from context should be able to tell the difference between Linus Torvalds and a guy who creates wordpress sites for his friends.
There are pure SAs around, and I don't just mean the caste system. The entire sysops organization was just one example. They may have been rolled up into another organization but the people still exist.
Although in the case of SQL, specifically, database administration has historically been a specialized sub-category of system administration.
Yes, probably. That doesn't mean your role on the project, in the company, or in your job description isn't "Database Administrator." Or operations engineer or reliability engineer or whatever fancy name you want to call the fact that you are more responsible for running the system than building it.
A more realistic example, though, is that a DBA and developer collaborate; the DBA collects and analyzes data from the production system and shares this information along with recommendations to the developer, who is ultimately responsible for restructuring the query.
In a very small team there's probably no need for these roles to be divided.
When I was interning at Google I noticed that these roles were differentiated as SRE and SWE. I didn't notice a lot of competency difference, just different application areas of the same competence. I think many SREs and SWEs could probably switch roles successfully.
I never had a google internship or anything similar, and my first technical job out of college involved supporting an internally developed, browser-based ActiveX/.asp/vbscript tool for viewing and annotating loan documents. For a long time we on the systems team were unable to upgrade IE on user workstations because of a critical, easily reproducible bug in the document viewer tool. Eventually, I managed to get access to a snapshot of their codebase and tracked down the bug in a few hours (despite never having written a line of .asp or vbscript in my life and not having access to any of the documentation for the 3rd party libraries and activex components). The devs, of course, didn't listen to me anyway, despite my basically handing a solution to them. The bug remained for another year until that section of code was incidentally re-written to support some other feature.
The next company I worked for, a web company, had a much better relationship between development and operations. It might have helped that we had a lot of Indian developers, who didn't seem to have the same prejudices that western devs do: that system administrators are people who just aren't skilled enough to do development.
tptacek's comments sum up the point - hence the qualifier of "pure". I think the SysAdmin role is rapidly evolving to look more like a programmer's job description from several years ago.
Sure, someone still has to write the puppet configs and monitor and manage the deployment, but that person is a far cry from the skillset and job description of 5 years ago.
My read is that he's saying that the days of the "system administrator" with a job description that begins and ends with tar flags and Apache vhost configuration are numbered.
He's not saying sysadmins are being replaced by Puppet and Chef; he's saying they're being replaced by a new kind of sysadmin that can be expected to extend Puppet and Chef.
This seems closely related to a point Patrick McKenzie makes all the time, that being the difference between a "programmer" and "someone who provides business value by means of applying specialized skills with programming languages". Don't be a "programmer".