News for hackers. Stuff that matters.
Re:Chef 11 released by lifeisstillgood on Sunday February 10, 2013 @03:05PM (#5197612)
Can someone explain to me the benefits of moving to chef or puppet as opposed to say my current approach of writing deploy scripts in pythons fabric (basically ssh and file operations library)

I have even used Puppet but I could not get over the feeling that it is more complex than I would imagine for small end users (I have managed upto 40 servers with my libraries - it seemed quite feasible to double that. Maybe it's me but 100 servers sounds a lot and is manageable (deployment wise) with controlled scripts.

Re:Chef 11 released by swalberg on Sunday February 10, 2013 @04:57PM (#5198109)
It's helpful to think of two separate processes: managing the server that will host your application(s) and deploying to the servers.

SSH + shell scripts are pretty good for one off things. But they're rarely idempotent. Chef and Puppet will say "you asked for nginx version x, you have it, so I won't bother installing it". You can also handle dependencies and roles, so you end up with things like "this hosts a Rails application, so you need Ruby and bundler, nginx configured a certain way, and the upstream load balancer needs to point to the new server". So using shell scripts ends up either taking longer or as you grow it gets more complex because you have to check for different things that a specialized tool just handles natively.

On the deploy side, ssh + shell scripts might be a better solution (IMHO stuff like Capistrano and Fabric can be seen as a fancy wrapper around that). So the procedure for a deploy might be "check out source, copy to a certain folder, create these symlinks, restart these daemons". Deploys, by their very nature, aren't idempotent, as each deploy is intended to change the state, you either finish it or abort and roll back.

I interviewed community managers from Puppet and Chef over at http://linuxadminshow.com, there are some interesting comments from both about the need for it even at small scale. Nathen from Opscode had a few comments about erchef. He also runs a podcast called "Food Fight" where they very recently talked about using Chef for the deploy process (http://foodfightshow.org/2013/01/application-deployment.html)

Re:Chef 11 released by bithive123 on Sunday February 10, 2013 @03:55PM (#5197857)
Configuration management is one of those things where you reap most of the benefit just by doing it, not by doing it with a particular tool.

That said, I prefer Puppet because its abstractions make more sense to me, and are pretty easily extended using Ruby.

Puppet is also trivial to bootstrap (at least on Debian). Unlike other Ruby stacks, I can install it using the package manager and be up and running in no time. With Chef you have to maintain additional services like CouchDB and RabbitMQ.

The real reason not to use ad hoc scripts or a tool you wrote yourself is that things like cfengine, Chef, and Puppet are maintained and used by communities -- your successor(s) will thank you.

Re:Chef 11 released by viraptor on Sunday February 10, 2013 @05:14PM (#5198186)
> With Chef you have to maintain additional services like CouchDB and RabbitMQ.

And care about a lot of dependencies. Installing chef-server on a fresh ubuntu quantal:

    0 upgraded, 444 newly installed, 0 to remove and 58 not upgraded.
    Need to get 280 MB of archives.
    After this operation, 775 MB of additional disk space will be used.
Of course a lot of that is due to how ubuntu packages java, but still... that's quite a lot of software to care about.
Re:Chef 11 released by jtimberman on Monday February 11, 2013 @10:23AM (#5201198)
With Chef 11 Server, the CouchDB component is removed for PostgreSQL. RabbitMQ is still there. Opscode provides a single package with everything above libc required to run the server.

http://www.opscode.com/chef/install - "Chef Server" tab.

If you'd like to build your own:

https://github.com/opscode/omnibus-chef https://github.com/opscode/omnibus-software https://github.com/opscode/omnibus-ruby

Cheers, Joshua Timberman, Technical Program Manager, Opscode, Inc.

Re:Chef 11 released by btmspox on Monday February 11, 2013 @12:10PM (#5201839)
A large part of that was an unfortunate side-affect of gecode (a library Chef used for dependency resolution in Chef 10) not having a -nox package, thus pulling in libqt4-dev, which then spirals out of control. The chef-server cookbook used the gecode cookbook which automated installing from source without qt to avoid this if one desired.

The Chef 11 Server doesn't use gecode.

Re:Chef 11 released by pjungwir on Sunday February 10, 2013 @06:56PM (#5198535)
> With Chef you have to maintain additional services like CouchDB and RabbitMQ.

You can avoid those dependencies by using Chef Solo instead of Chef Server. In my opinion chef-solo is also a much easier way to learn Chef, so you can get the main concepts down before tackling chef-server.

But yes, Chef's dependencies are staggering. I once tried using an AWS micro instance for a Chef server, and it kept crashing because the system didn't have enough resources.

Re:Chef 11 released by pedrocr on Sunday February 10, 2013 @03:23PM (#5197706)
Can someone explain to me the benefits of moving to chef or puppet as opposed to say my current approach of writing deploy scripts in pythons fabric (basically ssh and file operations library)`

I can't talk about chef but puppet basically gives you a declarative language to specify the end-state in which the server config should end up as. So you say something like "the apache server must be enabled and running" and then when the client runs it makes sure the state matches that. So if for some reason apache gets disabled (say a new version of the package disables it) the next time the client runs it will right that.

I have even used Puppet but I could not get over the feeling I was placing a lot of trust in an agglomeration of system in provided recipes

You don't have to use the recipes. I had the same worry so what I did with my own puppet usage was set a rule that I won't copy recipes from anywhere and just use puppet to do the same things I would do directly on the server but describe them using puppet so I can replicate them later. This has worked great for me. I still have to learn all the details of how to configure everything on the machine but I now have a version controlled log of how I've configured everything and I can very easily apply the same recipe between two machines.

And even though I don't use copied recipes puppet itself abstracts away a bunch of stuff. So for example "make sure apache is installed" is implemented with apt in ubuntu and rpm in redhat. So if I wanted to repurpose my recipes from ubuntu to fedora I'd have to recode paths and package names but I wouldn't have to figure out how to interact with the package manager, what the command to create a new user is, etc.

Re:Chef 11 released by viraptor on Sunday February 10, 2013 @04:51PM (#5198080)
With chef/puppet/salt/... it's easier to manage things that happen to change. For example it's much easier to express "generate configuration from this template into that file and if it changes restart following services" in those systems. That's what they're designed to do and that's what requires minimal number of steps.

With fabric it's easier to say - do this, do that. Both have their own place and you can use both for different tasks.

It may not matter at the first deployment stage (it's always install this, install that), but it does make a difference when you do continuous deployment for example. That are many dependencies that you'd have to implement in details and manually with systems like fabric.

Re:Chef 11 released by makerbreaker on Sunday February 10, 2013 @05:53PM (#5198338)
Another benefit is abstracting away the management via a boxed in webapp. I manage about 2000 servers all running chef. Things like IPs, host files, firewall settings etc, are stored in databags, each server has its own databag that can be manipulated by a Rails app/the Spice gem (thanks Dan Ryan!). As someone who is tasked with managing all of this, along with bringing the less experienced employees up to speed, it is a way for me to normalize all of these things, and abstract things away enough that almost anyone can open a port up on any server, setup a rails app etc etc. And whomever takes over after me, will just have to read one set of docs, get an overview, and they should be able to follow the design patterns of how things are setup, much more easily then random ssh/bash scripts.
Re:Chef 11 released by brugidou on Sunday February 10, 2013 @04:16PM (#5197941)
Chef is an automation framework that can handle your entire infrastructure. We use it to manage thousands of servers from PXE boot, OS installation and to configure the entire software stack. It basically give you a better environment to manage your automation scripts. The cookbooks are unit of works and you have testing frameworks.
Re:Chef 11 released by siliconc0w on Sunday February 10, 2013 @03:36PM (#5197784)
Chef and Fabric can play nice together. Use chef to configure the system but use fabric to manage them. Chef isn't fantastic at operational activities, especially those that need to be coordinated with a couple of nodes. Check out Pychef to drop your chef roles and hosts into fabric scripts.
Re:Chef 11 released by jvehent on Sunday February 10, 2013 @03:09PM (#5197632)
Chef maintains a database of all the nodes in your environment. Nodes can search for neighbors and configure themselves accordingly.

I wrote a firewall cookbook that takes advantages of Chef's search capabilities to dramatically reduce the maintenance of our firewalls, while keeping the rules ultra specific (inbound and outbound). Check it out: https://github.com/jvehent/AFW/ & http://jve.linuxwall.info/blog/index.php?post/2012/11/14/AFW

Re:Chef 11 released by lifeisstillgood on Sunday February 10, 2013 @03:14PM (#5197653)
I maintain a database of all the nodes (servers, Linux or BSD so not talking routers ) as well - one would go mad otherwise. And I feel that at certain scales one should tell hosts what to do, not rely on some auto discovery Process - especially if that process is not out and out cloning.

Edit: just to clarify I am not attacking your choice here or your contribution to the state if the art in GitHub - we need more people making. But I would like to understand why a guy who is unlikely to run more than a couple dozen servers again needs to learn to debug and battle something like chef when the imperative scripting approach is simpler and more direct

I suppose I am asking, at what scale does chef et al pay off compared to expect-like scripts (assuming the development time of a simple script is roughly equal in both)

Re:Chef 11 released by jvehent on Sunday February 10, 2013 @03:27PM (#5197729)
unlikely to run more than a couple dozen servers

Then, yes, Chef is probably overkill. Although, the resources (template, command, etc...) are nice and helpful. But I wouldn't replace an existing working process with Chef is there's no obvious benefit.

Re:Chef 11 released by btmspox on Monday February 11, 2013 @11:58AM (#5201771)
Do it early, for many reasons. I worked as a systems administrator for startups before we wrote Chef, and I wouldn't go back to managing infrastructure by hand.

A few reasons:

* You don't have to maintain that database manually, Chef collects and stores node information for you.

* If you've already automated building out your servers, you're all set when you need more of them.

* The recipes document your systems, so if you "get hit by a bus" someone else can figure out how the bits get around.

* When your datacenter burns down, you can rebuild quickly from bare metal, a chef repository, and backups of your application data.

Also see this article written by Jesse Robbins, one of the Opscode founders, but before Chef existed:

http://radar.oreilly.com/2007/10/operations-is-a-competitive...

Re:Chef 11 released by stevekemp on Sunday February 10, 2013 @04:54PM (#5198097)
> needs to learn to debug and battle something like chef when the imperative scripting approach is simpler and more direct

The biggest difference is, as the previous replies hinted at, Chef/Puppet/CFengine will each run "constantly" checking the state of the system against their rules.

With your fabric-approach you can only cause changes to happen when you're in front of the computer are you run "fab apply", "fab apt-get-upgrade", etc.

FWIW I also use a simpler alternative, recently documented, introduced, and explored here:

http://www.steve.org.uk/Software/slaughter/guide/

My system also relies on manual re-runs of the "slaughter" tool, but I schedule that with cron.

Re:Chef 11 released by tacticus on Sunday February 10, 2013 @05:56PM (#5198348)
Additionally these tools can take a host in an unknown state and turn it into what you want.
Re:Chef 11 released by stevekemp on Sunday February 10, 2013 @06:00PM (#5198361)
With the obvious caveat that you must have written the rules/policies/recipes for that to happen.

(I've seen some cfengine repositories, for example, that assume Apache is present and just start tweaking the configuration files. If you're doing it properly you'd want to define a rule initially "install apache". That's why it is often instructive to test out your rules on a pristine/virgin installation of your preferred distribution to see that it works from start to finish.)

Re:Chef 11 released by js2 on Sunday February 10, 2013 @07:10PM (#5198591)
You might want to take a look at Ansible. It purports to be simpler to get going with than Chef/Puppet.
Re:Chef 11 released by jvehent on Sunday February 10, 2013 @03:18PM (#5197674)
And I feel that at certain scales one should tell hosts what to do, not rely on some auto discovery Process - especially if that process is not out and out cloning.

That's one thing, in Chef, that scares me. Access control on node attributes is inexistent. Even in private/hosted chef. So that a node can modify its own run_list, its own tags, etc...

A frontend node that gets rooted can suddenly become a backend database. If the environment is configured to allow for automated configuration of everything, this can get ugly very quickly.

Even with Chef, human approval of new nodes and roles is still mandatory.

Re:Chef 11 released by idunno246 on Sunday February 10, 2013 @04:27PM (#5197978)
doesn't changes to run lists and roles require a pem you're supposed to delete immediately after registering a node with chef?
Re:Chef 11 released by viraptor on Sunday February 10, 2013 @05:01PM (#5198125)
validation.pem is only used for the first self-registration of a node. It doesn't play any role later on.
Re:Chef 11 released by bahman2000 on Sunday February 10, 2013 @04:53PM (#5198093)
I like that chef also knows a whole lot about my nodes, db is searchable and recipes can use search results.

I can do things like: "all my webservers with RAM > N should have different httpd.conf settings"