News for hackers. Stuff that matters.
Re:Puppet 4 is now production-ready by duggan on Thursday April 16, 2015 @03:08AM (#9386342)
Automation is my day job and I'm still not convinced any of them is a significant improvement over a well maintained set of shell scripts and Makefiles.
Re:Puppet 4 is now production-ready by kolev on Thursday April 16, 2015 @03:28AM (#9386401)
Same here. I recently found out about Babashka [0] and Declarative Bash [1], but I've written something similar already. It's much easier to find talent that knows Bash and can learn a readable library of Bash scripts within a day vs finding talent that knows the flavor of the day (every year people are migrating from the old favor to the new - Bash => CFEngine => Pupper => Chef => Salt => Ansible => ... => Bash).

[0] https://github.com/richo/babashka

[1] https://github.com/threatgrid/declarative.bash

P.S. I've recently been fascinated by Rex [2] (They should fix their text logo though - I find it terrible).

[2] http://www.rexify.org/

Re:Puppet 4 is now production-ready by e12e on Thursday April 16, 2015 @04:17AM (#9386563)
First I thought this was some dialect of Rexx[1], but then I saw: "Easy to learn, it's just plain Perl". A well. No thank you, I'll stick with ansible. Not that I think Rexx would be a good idea either, it just reminded me of ARexx for AmigaOS...

[ed: forgot to ask: you find it easy to find talent that can not only learn some bash script library, but write new code without introducing some gaping hole somewhere?]

[1] http://www.rexxla.org/

Re:Puppet 4 is now production-ready by kolev on Thursday April 16, 2015 @02:36PM (#9389923)
I'm sorry, but with Rex you don't use much of Perl. It's a DSL, which is based on Perl.
Re:Puppet 4 is now production-ready by crdoconnor on Thursday April 16, 2015 @03:51AM (#9386476)
IMHO puppet isn't significantly better than bash scripts. There are some improvements, but it has significant drawbacks too.

Ansible is way better though, mainly due to:

* Idempotence

* Templating

* Being declarative (turing completeness in a language = more likely to be buggy)

Re:Puppet 4 is now production-ready by e12e on Thursday April 16, 2015 @04:13AM (#9386550)
I agree with you, but re: templating: if you're using shell scripts, why not use m4 as well?

Shell of course has templating -- but it's a little dangerous...:

  template=teapot size=short build=stout \
  cat <<eof > teapot.conf
  I'm a $template - $size and $build. I live at $(hostname).
  eof
[ed: m4 is a little like cpp (the c pre-prosessor) on steroids (or acid). To get a feel for how it relates to templating for config files etc, see eg:

http://box.matto.nl/m4.html ]

Re:Puppet 4 is now production-ready by mercurial on Thursday April 16, 2015 @04:42AM (#9386643)
At first glance, I don't see that m4 does anything jinja2 (Ansible's templating system) doesn't do better.
Re:Puppet 4 is now production-ready by e12e on Friday April 17, 2015 @07:10PM (#9397696)
As I mentioned in another comment, I'm not suggesting using m4 with/in-place of ansible -- rather as a compliment to the shell. I suppose one could use jinja2 stand-alone with the shell -- but then you get the python dependency etc... I meant that m4 might be a better fit to go along with bash/ksh.

Eg:

  dpkg-query --show \
  --showformat='${Package;6} ${Installed-Size;4}^L\n' \
  m4 python2.7-minimal dash bash mksh ksh \|
  sort -n -k2

    dash  212
      m4  382
    mksh  878
     ksh 3110
    bash 3941
  python 5456
(Not that you can avoid python-minmal on Debian - but in a chroot you might, etc).

As for can and cannot do, consider this "feature" of m4:

  $ cat <<eof|m4 -Dme=you -Dpredictable=surprise
  me=predictable
  eof

  you=surprise
Again, I'm not suggesting m4+dash is a great idea vs ansible -- just clarifying a couple of things...
Re:Puppet 4 is now production-ready by 23david on Thursday April 16, 2015 @04:53PM (#9390683)
If we're discussing templating languages, I definitely prefer Mako over Jinja2 and think it's much more appropriate for devops templating. But Jinja2 is the default in a lot of projects, including Saltstack and Ansible, and it's hard to fight the currents all the time.
Re:Puppet 4 is now production-ready by mercurial on Friday April 17, 2015 @01:08AM (#9392346)
What do you like more about Mako? I haven't used it in years, but I remember it being fairly similar in capabilities to Jinja2.
Re:Puppet 4 is now production-ready by 23david on Friday April 17, 2015 @07:49AM (#9393663)
Not sure if it's a good idea for me to wade into that particular religious war... :-)

Some good comments on both sides here:

http://www.quora.com/Python-Web-Frameworks/What-are-the-adva...

http://www.reddit.com/r/Python/comments/ktjm2/jinja2_or_mako...

http://stackoverflow.com/questions/3435972/mako-or-jinja2

Re:Puppet 4 is now production-ready by e12e on Thursday April 16, 2015 @05:43AM (#9386798)
If you're going to use shell and not ansible, then why not use shell with m4. I'm not suggesting shell+m4 is better than ansible, or that ansible+m4 (wtf?! right?) would be a good idea.

Just why not use a readily available template/macro language along with shell scripts, if you've decided to use shell scripts.

I suppose you could use jinja and bash -- but why pull in two full interpreters as a dependency?

Re:Puppet 4 is now production-ready by crdoconnor on Thursday April 16, 2015 @04:55AM (#9386684)
>why not use m4 as well

* Because jinja2 is overall a better templating language.

* YAML (not turing complete) + templating = readable and clear (this is how ansible does it).

* Bash (turing complete) + templating = unreadable and messy.

"I want my configuration code to look as clear and readable as my C++" <-- not a great sales pitch :)

Re:Puppet 4 is now production-ready by 23david on Thursday April 16, 2015 @07:18AM (#9387113)
Ansible is a good simple tool with a small learning curve so it's not very intimidating for developers. But it has its strengths and weaknesses just like everything else. Puppet, Chef and Saltstack are all fantastic tools. It all depends on how you use them.
Re:Puppet 4 is now production-ready by brandonmenc on Thursday April 16, 2015 @04:14AM (#9386553)
Aren't those also features of Puppet?
Re:Puppet 4 is now production-ready by crdoconnor on Thursday April 16, 2015 @04:47AM (#9386657)
Yes, but puppet has other issues. You don't have to mess around with certificates, master servers and extra services with bash. You do with puppet.
Re:Puppet 4 is now production-ready by brandonmenc on Thursday April 16, 2015 @05:03AM (#9386699)
So why ansible? I use puppet. A quick glance at ansible suggests that all of that (certs, servers, services) is replaced in ansible with SSH, right? But you still need to manage keys. What am I missing?
Re:Puppet 4 is now production-ready by crdoconnor on Thursday April 16, 2015 @07:26AM (#9387145)
You would need to manage keys whatever you did. Puppet just means that you have to manage keys AND certs, so you have double the headache.

Puppet also means you have to babysit the client service and the puppet master, and if they go down, you have to get them back up again. I deal with this problem a lot and I wish I didn't have to.

Re:Puppet 4 is now production-ready by berkes on Thursday April 16, 2015 @03:16AM (#9386368)
Automation is not my day job, and I am very happy that there are frameworks (I work with Chef) with communities that offer standards, guidelines and libraries for a well maintained set of shell scripts and Makefiles. Instead of having to build and maintain them all myself.
Re:Puppet 4 is now production-ready by duggan on Thursday April 16, 2015 @05:30AM (#9386759)
I agree that having standards and best practices is a boon, especially if they can be packaged up in an easy to use way. Like anything, Chef brings its own set of idiosyncrasies to the table. However, I don't like how it brings all of Ruby's idiosyncrasies along for the ride too.

If you're not immediately comfortable and familiar with both ruby and the quirks of its ecosystem, you end up investing a lot of time into getting a development environment up and running (and having it reproducible by other members of your team).

Most infrastructure tools, if they're any use, are rapidly evolving. Conway's law applies here, and the maintainability of Chef mirrors that of its ruby ecosystem roots - people tend to freeze dependencies rather than upgrade. That's an antipattern we don't need in infrastructure.

Honestly I just think we can do better. For me, I'm not sure (read: I really don't know, I'm not being facetious) the costs outweigh the benefits, but for someone totally new, perhaps it helps them get stuff done faster.

Re:Puppet 4 is now production-ready by otterley on Thursday April 16, 2015 @05:03AM (#9386703)
I've sadly learned that given enough time, you'll likely end up mostly using your own set of private cookbooks, even for configuring things community cookbooks already exist for. The quality of the community cookbooks varies wildly from "pretty good" to "toxic waste," and the likelihood of getting fixes upstream fast enough to reflect back into one's own environment is generally low, even for good maintainers.

And Chef's strict adherence to X.Y.Z semver for cookbooks (even though cookbooks don't fit semver well) and inability to declare "my cookbook foo_prime can be used whenever cookbook foo is called" makes it near impossible to maintain your own branch while you wait for your fixes to make it upstream.

This makes it difficult to try to get things done when you encounter a cookbook that just doesn't work quite right. The cookbook ecosystem is not yet mature enough at this point in time to rely on.

Re:Puppet 4 is now production-ready by e12e on Thursday April 16, 2015 @04:01AM (#9386513)
Right. But half-way, somewhat maintained ansible is much better than the similarly maintained shell scripts and Makefiles...

And you get somewhat more portability with it.

mattjaynes mentioned his blogpost in a previous discussion on this: https://devopsu.com/blog/ansible-vs-shell-scripts/

I think it illustrates the difference well. I agree that you probably could do much of the same with discipline, source control, shell scripts and Makefiles. But there's a reason why we use other languages than shell scripts. Shell scripts are finky, and it takes years for people to stop making "rookie" mistakes. If they ever do. "[" vs "[[", ${var} vs $var, ksh vs bash... the shell gives you way too much rope (and I like shell scripts :-).

Re:Puppet 4 is now production-ready by duggan on Thursday April 16, 2015 @04:49AM (#9386662)
I should probably couch this in the caveat that I am using Packer and Terraform to control the infrastructure itself :)

So machines are effectively configured "offline", tested, and replaced. It's not enough though (usual things like databases and development tweaks like config changes are problematic) so I'm back into the world of config management to see how it can help.

Matt makes some good points, and we broadly agree on controlling infrastructure, I just think that most of the CM systems are inadequate abstractions - they don't go far enough, and it leaves them largely as convenience wrappers around shell, but ironically enough, usually without the debuggability of shell.

I've worked with Chef for years, and recently have started seeing what I can do with Ansible. For me, the major thing in Chef's favour was the combination of attribute driven configuration and integration with things like Test Kitchen and ServerSpec.

Chef requries buy in to the ruby ecosystem though, and that's just not something I want to expose non-ruby developers to (it's just too brittle), which is why I'm exploring Ansible. What I'm finding, though, is that for any reasonably interesting tasks, I'm still forced to execute "command" strings, which is not an improvement over shell, and gives people the false impression that it's idempotent (or even just safer).

Also most of these systems are trying to upsell me a centralized management system and UI, handing over the keys to my infrastructure, which is just not something I'm willing to do.

Edit: I'd also add that cross-OS portability of CM (if that's what you're referring to), in my experience, a bit of a red herring.

Re:Puppet 4 is now production-ready by skuhn on Thursday April 16, 2015 @05:36AM (#9386779)
I think the biggest issue with current CM systems is that they aren't made by people with any experience actually administering large scale infrastructures.

The focus to me seems all wrong. Ansible has a blue million modules, but so many of them are incomplete or downright useless. The cron module can handle cron.d files, but it still tries to manage individual lines in the file and tracks state with comments. It can easily get confused and add multiple entries for the same cron job. Puppet had the same problem. The solution is obvious: just make it barf a pre-made file into /etc/cron.d. But why does no CM actually do this?

Over the last year I have spent a lot of time using Ansible, and I've eventually given up on the cron, sysctl, service and get_url modules and resorted to running commands or copying files. This isn't very clean and it can be brittle, but the alternative of fixing the modules is off the table. The debugging or logging support in most modules is literally non-existent.

There also seems to be a pretty serious scaling issue with the way Ansible processes its inventory, even when using the (absolutely essential) _meta optimization that provides the entire inventory in one command. This is at only a few hundred hosts and tens of thousands of variables.

Contrast this with Puppet: while there are a great many things I hate about it, the dependency system and ability to define your own types inside of the DSL was pretty powerful. Too bad that you eventually had to get down to writing Ruby modules to do certain things. Also too bad that they built it as a client-server system, allow you to export resources between hosts, and took forever to make the execution order deterministic.

The final nail in the coffin for CMs and me is that Ansible (along with Chef, Puppet, Salt) try to do so much but have no concept of absolute state enforcement. The host should always either be in state A or B; the CM system should be able to move in either direction between A and B; and it should never be left in a state inbetween A and B. With Ansible it's quite possible to get permanently locked into a middle state, since it may die before handlers are run (which are then never triggered in subsequent runs).

Building these tools is admittedly harder than it looks, but the fundamental building blocks of a CM system aren't that tough. So many recipes boil down to literally a three step process: install package, copy config file (templating is super nice), start service. Stop trying to play nice with local modifications, the whole point is that I want you to enforce state absolutely -- local modifications are the enemy!

I wish there was a CM that focused on getting this stuff absolutely right and then made it easy to build on top of that.

Re:Puppet 4 is now production-ready by helloiamaperson on Thursday April 16, 2015 @07:21AM (#9387120)
Take a look at bosh: http://drnicwilliams.com/2012/05/15/bosh-what-how-when/ . It has its own problems, but it does exactly what you want, and is used for large scale deployments.
Re:Puppet 4 is now production-ready by skywhopper on Thursday April 16, 2015 @09:50AM (#9387871)
Automation is also my day job. I've got a ton of shell scripts and a ton of Ansible playbooks. No reason they can't work together.

For some tasks, shell scripts are better/clearer/easier, and you can call those scripts from Ansible. For other tasks, Ansible provides better tools, and hey, you can call Ansible from your shell scripts!

I've added a few ansible-friendly options to some of my shell scripts like adding an --ansible-mode flag that then outputs just a one-line "OK" or "Changed" message that then I have Ansible parse to determine if something changed during the run. Or if you're more ambitious you can just turn your shell scripts into Ansible modules by having them take arguments in a slightly different way and output JSON.

A free templating system (Jinja2) and cross-platform capabilities are nothing to shake a stick at either. Not to mention, Ansible does a good job of helping organize your hosts and parameters in one place, and forces you to modularize your "scripts" (in the form of playbooks and roles) in a beneficial way.

It's definitely worth a look. You can start small, and use it only for one thing, and grow from there. That's one of its biggest benefits over Puppet and Chef, which don't really have a way to go small-scale.

Re:Puppet 4 is now production-ready by duggan on Thursday April 16, 2015 @01:15PM (#9389323)
I do like the "mass SSH" functionality that Ansible provides, it's something accomplished years ago with Capistrano, but I certainly prefer having it in Python (now that I've wrangled together a dynamic inventory interface to Terraform's tfstate files).

Shell is not what you want to go to for orchestration, but, essential as it is, I think it's a separate concern from managing the configuration of the machine.

You're absolutely right about being able to start small as a major point in its favour, though. Not enough infrastructure tooling accomplishes this.