News for hackers. Stuff that matters.
Re:Why Chef? by apinstein on Sunday October 16, 2011 @10:45AM (#3117402)
I am a co-founder of a small SaaS company. We recently decided to make the investment of upgrading our infrastructure setup process from "Hey David go do that" to being 100% chef-based.

I managed the process, and consider it a success. However, here are some points I would make:

1. It took a long time. Let's be generous and say it took 2-3 man-months of time to set up 4-5 different projects and roles. This was probably 10-20x what it would've taken to set up the servers directly. Why? Learning curve with chef for both our programmer and sysadmin. Figuring out how to make config changes automatable and idempotent.

2. The scale you get from chef is bigger than managing production infrastructure. We now use chef for not only production deployment, but also dev. Once paired with Vagrant, we are able to get new devs up with a complete stack in about 10m of keyboard time. If we need to upgrade to some new version of something, only one person has to deal with the sysadmin; everyone else can just update their box.

3. I think it will save money in the long-term. A good sysadmin is $100/hr+. Unfortunately you have to pay that rate whether they're doing architecture, security review, or just editing text files. With chef, a non-sysadmin resource can generate recipes with just architectural advice and review from a sysadmin. This is much more efficient, especially for small shops where a sysadmin is an expensive and not immediately available resource.

Re:Why Chef? by tptacek on Sunday October 16, 2011 @11:57AM (#3117549)
A good sysadmin costs $200k/year? Or do you have a contract sysadmin? Because that number sounds way, way, way high.

(We recently hired a sysadmin "+").

Re:Why Chef? by apinstein on Sunday October 16, 2011 @12:19PM (#3117588)
We've been in business for about 8 years and use about ~100 sysadmin hours a year.

I interviewed several sysadmins over the years and no one that I deemed competent charged less than $100/hour.

I did find people charging as low as $50/hour but they didn't seem that great, or that reliable, or that available.

For me as well, you pretty much need to trust your sysadmin more than almost anyone else in your company (except people that can sign checks).

Trust comes at a premium.

I will yield that I don't like working with substandard people. I hate having to manage people and am willing to pay a premium for people I trust to work on the right things with the right skills in a timely manner. Besides, it doesn't scale. One of my business goals is to never have middle management.

Re:Why Chef? by tptacek on Sunday October 16, 2011 @02:18PM (#3117880)
You'd be more convincing if you didn't imply that anyone who worked for less than $100/hr was "substandard". For full-time salaried, I assure you, $200k/yr is not the going rate for sysadmin.
Re:Why Chef? by nestlequ1k on Sunday October 16, 2011 @02:25PM (#3117897)
You cant directly equate contract hourly rate with yearly salary. If you don't have the budget to hire a 100k a yr salary, you'll have to pay contract rates which could be over 100/hr
Re:Why Chef? by tptacek on Sunday October 16, 2011 @02:29PM (#3117907)
I buy that the valley is so hot right now that a sysadmin commands $100k/yr there, but they don't in Chicago, Seattle, or New York.

I'm not looking to argue so much as to inject some more data into the price point that was casually dropped on this thread earlier. I do not think the other guy overpaid for sysadmin; if he's got an amazing admin, great! I can see paying a premium for that.

Another point I'd like to raise is, if you're paying $100k/hr for sysadmin, and using them frequently, contracting instead of fulltiming sysadmin might be penny-wise-pound-foolish. But maybe not, if you're only paying $10,000/yr in sysadmin. We've used over 100 hours of admin in just the last couple weeks. A great hire; one we made "37signals-style", after realizing that doing all the sysadmin chores ourselves was subtly making us all miserable and ineffective.

Believe me, I know the difference between contracting rates and salary. ;)

Re:Why Chef? by apinstein on Sunday October 16, 2011 @02:56PM (#3117971)
I don't think I implied that. I said only that I interviewed a handful of people and of that sample set, the people that charged less than $100/hour were not competent. They suffered from things like only knowing one flavor of UNIX, or having never heard of chef/puppet/etc, or not really knowing shell scripting, or not having heard of security stuff that even I know about.

I am not an experienced sysadmin, but when I feel like I could do a better job than the person I'm interviewing, I consider that substandard.

Re:Why Chef? by illumin8 on Sunday October 16, 2011 @02:35PM (#3117918)
He is absolutely correct. A senior Linux admin is going to charge $75 an hour, and the contracting company that finds them will charge $25 an hour for handling the paperwork.

You could hire someone full time for about $120K a year but then with benefits and employer taxes it will end up being closer to $150K a year anyway.

Re:Why Chef? by tptacek on Sunday October 16, 2011 @02:43PM (#3117932)
$120k is nosebleed high for full-time sysadmin.

Are you sure we're talking about "system administrator" and not a more specialized role like "devops"? I know to us nerds those are basically the same thing, but they really aren't. If you're doing sysadmin for the primary purpose of deploying and maintaining boxes designed to run one proprietary application, you might be a devops person, and not a sysadmin.

Again, this might be a valley thing. I've got a bunch of friends who have complained how hard it is to get sysadmin in SFBA. Just know, if that's the case, the dropoff in salary outside the valley is waaaaaaaaaay sharper than it is for dev, which is pretty much just COLA adjusted from place to place.

(We staff offices in Manhattan, Chicago, and Mountain View, for what it's worth).

Re:Why Chef? by apinstein on Sunday October 16, 2011 @03:00PM (#3117976)
For my part, I suppose I was talking devops, or at least a sysadmin that thinks devops is a good idea and can help make it happen.

It's probably wrong from a larger perspective to lump the two together, but for me sysadmin as a role has been supplanted with devops. If you are a sysadmin that doesn't want to automate away their work via devops, you are not doing it right.

Re:Why Chef? by tptacek on Sunday October 16, 2011 @03:26PM (#3118055)
I agree that as a career choice, "system administrator" seems less lucrative than "devops". Remember, though, that not every sysadmin job is for an agile software shop. There are plenty of places that care a lot more about patch cycles than they do about single-click deployments.

More importantly though, bill rates and capability do not track each other. Bill rates track risk. There is a lot of amazing- but- unproven systems talent out there, and there are some very expensive pikers on the market too. Generally I would agree that the more you pay, the less likely you are to have to fire 3 months later.

It helps a lot to be able to do the job yourself, soup-to-nuts, so that you'll have a better shot at screening candidates.

Re:Why Chef? by illumin8 on Monday October 17, 2011 @11:03AM (#3120733)
Not sure where you're getting your numbers at, but $100K is entry level for a sysadmin in NYC, and $120K is normal for a sysadmin with a few years of experience. I'm sure northern California is similar.
Re:Why Chef? by tptacek on Monday October 17, 2011 @11:52AM (#3120975)
$100k is not entry level for a sysadmin in NYC. I get my numbers from the fact that just about half my company, including its headquarters, is in NYC.

I am absolutely confident that there's some role/vertical definition you can come up with where sysadmins are making $100k in the door. But I think you're discounting most of the market for sysadmins, just like the people on the other current HN sysadmin thread who all seem to believe that sysadmins also optimize SQL queries, fix C code, and teach systems programming to stupid developers.

Re:Why Chef? by Dalves on Sunday October 16, 2011 @05:20PM (#3118338)
You've just echoed my experience with chef but I'm still on the fence as to whether I think it was a success or not and whether I'd use it again. We were able to achieve everything we wanted but to say it took a metric fuckton of time would be a grave understatement. Even what are ordinarily trivial things took a lot longer than I expected. Some things I just found to be oddly designed and surprisingly inflexible, especially the recipe construct. I still to this day see no point of having a folder with 2 subfolders with 3 files combined for what amounts to "apt get install somepackage". And having to learn a DSL to replace the already very effective bash is in my opinion not a very enticing trade.

Although it's very new, I think I'll try out juju.ubuntu.com for my next large deployment instead of sticking with chef or puppet, if only for the flexibility of the hooks-in-any-language thing and a chance to feed my bash fetish.

Re:Why Chef? by lkanies on Monday October 17, 2011 @05:27PM (#3122539)
(disclaimer - Puppet Labs employee)

2-3 months is a long time.

We consistently hear 2-6 weeks to get to full production, with 30-120 minutes for some proof of concept work.

And one of the great things about Puppet is you can start very small - automate the parts that are hardest, most critical, etc., and do bits manually as it makes sense for your problem set.

And yeah, managing dev and prod the same way is absolutely critical for clean deployments.

Re:Why Chef? by devmach on Sunday October 16, 2011 @11:11AM (#3117453)
Is there a something like Foreman[1] for Chef ? Something able to create and spin up new VM and execute receipts after install ( by using just a "web browser" ).

[1]: http://theforeman.org/projects/foreman/wiki/Screencasts

Re:Why Chef? by apinstein on Sunday October 16, 2011 @12:22PM (#3117600)
Oh nice I hadn't seen Foreman before. Looks like EC2 console but for your own stack. We use XEN, that might be nice.

I don't know what you mean by "execute receipts" however in the video they kick off a puppet run, which makes me think you could similarly kick off a chef run instead.

Re:Why Chef? by manvsmachine on Sunday October 16, 2011 @12:49PM (#3117664)
I'm guessing that 'receipts' was meant to be 'recipes'.
Re:Why Chef? by devmach on Sunday October 16, 2011 @05:20PM (#3118335)
Sorry, my bad. I meant to say "recipes".
Re:Why Chef? by bryanwb on Sunday October 16, 2011 @10:50AM (#3117414)
awesome comments, apinstein

btw, I am copying your method of managing dotfiles with a Rakefile as we speak.

I counter that while it takes a while to learn chef, it also takes many times more effort to maintain a custom set of shell scripts over the long term

Re:Why Chef? by sjwright on Sunday October 16, 2011 @04:33PM (#3118226)
I learned a new word today: Idempotence. Thank you.