News for hackers. Stuff that matters.
Re:ShutIt – Automation framework for programmers by zwischenzug on Sunday July 10, 2016 @10:09AM (#12065616)
Author here. Quite surprised to see this.

Another related tool I'm working on is:

https://github.com/ianmiell/shutitfile

Why this over Ansible? It's a long story :)

Mainly because not everyone is a great or even decent programmer, so I wanted an easy way in for them for the situation I was in. Hence automation for everyone. But I don't think this is a serious competitor for Ansible.

Happy to answer questions as they arise.

Re:ShutIt – Automation framework for programmers by zwischenzug on Sunday July 10, 2016 @10:13AM (#12065626)
There are some videos here:

https://www.youtube.com/watch?v=zVUPmmUU3yY https://www.youtube.com/watch?v=UGp16yvSrFE

And I've blogged quite a bit about ShutIt use cases here:

http://zwischenzugs.wordpress.com/

and the genesis of ShutIt and Docker:

https://www.youtube.com/watch?v=-qHwL8C9UoA

Re:ShutIt – Automation framework for programmers by themckman on Sunday July 10, 2016 @10:50AM (#12065750)
> Why this over Ansible? It's a long story :)

As someone who just spent the last year of his life using Ansible, something tells me I already know this story. I feel for you.

Re:ShutIt – Automation framework for programmers by Rafert on Sunday July 10, 2016 @11:26AM (#12065893)
As someone who hasn't done much with it yet, care to elaborate? It seemed easier for me to grok than Chef.
Re:ShutIt – Automation framework for programmers by Azeralthefallen on Sunday July 10, 2016 @12:34PM (#12066137)
My life resolves around Ansible and has for the last year and a half. Ansible is a great tool, unfortunately it has a bunch of short comings. Especially now some of these are becoming a major issue.

- Its dependence on Python 2.X, unfortunately all the newer OS's have Python 3.X installed by default. While i have a role for installing this manually it isn't super nice.

- My main devops project consists of 11 different environments, and it manages nearly 2k servers. Trying to write environment agnostic roles is an absolute nightmare. Furthermore my group_vars, host_vars, and dynamic inventories have turned into spaghetti. Why? Because of how confusing variable precedence becomes, when you get into extremely complex hierarchy of group_vars which can randomly break.

- It is really hard to handle cases where playbooks/roles fail and deal with it cleanly. Blocks have helped a bit, but there is no way to grab the exception that caused the playbook to fail.

- The yaml syntax becomes messy fast, Jinja2 is powerful but there are times i wish you could just shove a single line of python into places to prevent the huge mess of with_items, register, set_fact, etc.

- AWS modules are a very mixed bag. A lot of them are missing features, and there is some very weird issues that apparently boto is the cause of (e.g. they should move to boto3). Even then i need to do a huge amount of boilerplate to launch ec2 instances.

- There is no way to share a common group_vars between inventories (all is specific to inventory). Which leads to a lot of duplication. (Yes the symlink trick does work, but causes a bunch of other issues).

I still love Ansible and i would pick it over Chef/Puppet. Salt is also another decent one that has made a lot of progress lately.

Re:ShutIt – Automation framework for programmers by zwischenzug on Sunday July 10, 2016 @12:46PM (#12066172)
That's interesting - I just received a request for ShutIt to work with Python3.

Regarding failure, ShutIt drops to a shell on error (if there's a tty), so you can correct manually before continuing (pause_point). These can also be triggered directly for debugging purposes.

I don't like YAML for anything complicated. The bash scripts I saw embedded in YAML felt wrong. Why not just use shell?

Env-agnosticism (eg 'install' packages rather than yum/apt) was built in because it was created in a bi-OS env (ubuntu and CentOS).

Re:ShutIt – Automation framework for programmers by gtirloni on Sunday July 10, 2016 @01:32PM (#12066335)
We rely heavily on Ansible and your description seems very accurate. I worry we have too much invested at this point to consider anything else. Still would pick it over Chef/Puppet.

Salt does look interesting but it was too young when we started. I don't know if we would be trading Ansible's shortcomings for Salt's.

Re:ShutIt – Automation framework for programmers by ams6110 on Sunday July 10, 2016 @01:26PM (#12066312)
I use ansible to manage several hundred servers. So not your scale, but I've felt some of this.

I stopped trying to write completely agnostic roles. This does result in some duplication but much less hair-pulling overall.

Yes ansible is what made me grow to dislike yaml.

Re:ShutIt – Automation framework for programmers by shlant on Monday July 11, 2016 @12:55AM (#12068975)
As someone who has previously used Chef and is currently using Ansible, I can understand those pain points. For me it's not as much of a problem as I have 10x less servers, but I have heard a lot of complaints about Ansible scaling. I definitely miss being able to just use Ruby in a cookbook, but it's usually very rare that I can't find a reasonable workaround.

Usually I describe Ansible as the tool for people who don't come from a programming background and won't be administrating a huge amount of servers. Otherwise, if you need scale then use Chef (as why would you want to have to deal with scaling AND Puppets terrible DSL).

Re:ShutIt – Automation framework for programmers by chucky_z on Sunday July 10, 2016 @01:46PM (#12066392)
Regarding variable precedence, it's pretty simple to grok in 2.x+, and well-documented now (it was a serious nightmare, and a very valid complaint... but isn't as much anymore).

For sharing common group_vars, what I do is my inventory passes an environment-aware variable which I include in the playbook by doing like:

`- include: "env_vars/{{ environment }}/main.yml"`

I feel I sidestepped some of the AWS modules issues simply by using Terraform. I still build all my images in Packer using the Ansible builder though.

Re:ShutIt – Automation framework for programmers by Azeralthefallen on Sunday July 10, 2016 @04:07PM (#12067059)
Unfortunately the above way will load the vars at a higher precedence then group_vars. Also if you have multiple groups with the same var but differing values you will need to start loading multiple files (or you can do |default, and use an alias to the value in group_vars (which leads to even more spaghetti)).
Re:ShutIt – Automation framework for programmers by rhizome on Sunday July 10, 2016 @04:45PM (#12067193)
Ever consider CFEngine?
Re:ShutIt – Automation framework for programmers by themckman on Sunday July 10, 2016 @02:18PM (#12066546)
Azeralthefallen hit the nail on the head for the most part and I wasn't managing anywhere near as many servers s/he was. I had come from Salt prior to Ansible and there was a lot to miss.
Re:ShutIt – Automation framework for programmers by zwischenzug on Sunday July 10, 2016 @10:33AM (#12065696)
Also, I'm looking for contributors.

If you have an interest in usable automation, an expertise in terminal internals, or just want to get involved, please get in touch.

@ianmiell

Re:ShutIt – Automation framework for programmers by blacksmith_tb on Monday July 11, 2016 @01:22PM (#12072624)
Since configuration management isn't a typical part of my day job, this is appealing - I have used Chef and Ansible a little, but haven't needed them badly enough to really choose one, which makes ShutIt attractive to me, especially recording sessions. This may be a 'careful what you wish for' but I am curious if you think it would be possible to parse the shell history file on a box to produce a recording retroactively? It would probably have to be interactive, press return to add this command or space to ignore it etc., but it would provide for backing up everything you'd done to a server, which I have certainly wished I had an easy way to do (short of manually poring over my history and cherry-picking the things that mattered).
Re:ShutIt – Automation framework for programmers by zwischenzug on Tuesday July 12, 2016 @11:25AM (#12079795)
Yes that's somewhat possible, but YMMV. Contact me if you want via Github (ianmiell)
Re:ShutIt – Automation framework for programmers by was_boring on Sunday July 10, 2016 @10:57AM (#12065771)
Does this provide automatic subcommands? For example if I write a module/script can I just do `shutit <custom command here>` from within any directory?

The use case is a heroku/docker type cli without needing to mess with the Python args module and setting up a bash alias

Re:ShutIt – Automation framework for programmers by zwischenzug on Sunday July 10, 2016 @11:02AM (#12065797)
Interesting idea - if you give me a specific use case (github issue?) I'll work on something.
Re:ShutIt – Automation framework for programmers by y4mi on Sunday July 10, 2016 @03:17PM (#12066817)
more like "why would you use this instead of fabric" [0]... ansible has a different focus, but fabric looks pretty much the same to me

[0] http://www.fabfile.org/