I've used Ansible to manage thousands of baremetal servers, you do have to build some stuff yourself but it mostly just works. The Ansible framework for gathering facts is pretty neat, it has a framework to do X or Y depending on facts from hosts, you can run it in pull model or push model, through AWX or jenkins. A lot of freedom.

If you want something that just works automatically and is free, well, I don't think you will find. Learning Ansible (or another configuration management tool) well will pay dividends through a lot of your career tho.

> I hear this a lot, but do they really? As best I can tell they require writing everything from scratch. I went down this path just an exercise to see what end-users were having to deal with. Everything in ansible appears to be re-inventing the wheel.

Other people's Ansible playbooks are often nice, because people mostly only publish things that stick to "best practice". I'll often use these if the exist and seem well thought out (and can do what I need done).

For things that there isn't an obvious "good choice of existing playbook", I'll sometimes write a "proper" one (we've got a quite good one here that got used a lot when we were deploying a lot of fundamentalist similar Grails/Tomcat/Apache apps with a bunch of common dependancies), or I'll just resort to using Ansible to run remote commands over ssh. If I know how to do it on the server from the command line, it's trivial to do exactly the same thing using Ansible from my laptop (or our bastion or config host). The trick here is making sure you don't write Ansible/ssh that screws up if you run it twice - which mostly isn't too hard to avoid so long as you remember to do so.

> I think what OP is asking for is some intelligence to automatically discover. > If Ansible can do that, I'm all ears, but I haven't figured out how.

I suspect this is a difference in approaches. It's easier in the cloud/vm world, bit I still treat bare metal servers more like cattle than pets. If I get a problem like "OS that doesn't get corrupted from unexpected poweroffs or permanently cut itself off from the network because of a bad config." I just get the box reimaged to a known state, then run the Ansible "update and deploy/configure from scratch" on it.

I don't think trying to build something to "intellegently auto discover" and repair a corrupted OS from hard powerdowns or a totally botched configuration change is a good use of my time... I'll just stand up another server from scratch using tried/tested automation.

Something else to consider is all the edge cases you'll hit with tools like Ansible and Puppet. I'm pretty sure neither of them handle apt getting into a locked state because an operation was interrupted. So you end up having to write some hacky script to detect that and fix itself.

Ansible is not even an option for IoT because it doesn't scale well and requires a stable connection. Puppet's agent model works better for large fleets especially if you're crossing multiple gateways.

No luck with Ansible pull? Sounds like it maybe ideal for your usecase. Git pulls are often incremental and compressed, and going over https may prove to be more stable than ssh over unstable connections. Since Ansible is executed locally, theres no need to have a persistent ssh connection to run your playbooks.

Ansible-pull doesn’t require a stable connection. It’s just a way to download a playbook and execute it unattended.

Ansible can also run in pull-mode, just like Puppet. Is there anything that hasn't scaled well that you could share?

Neither Ansible nor Puppet takes 6 months to configure. Yes you have to write the code (and/or pull pre-existing modules) to make the config changes you want, but that's the point of them: they are there to automate the changes of config on servers. If you don't want to change the config of a base install then there is nothing to write. And given config is going to be specific to your environment then of course you're going to need to customise your config management code. No two companies are going to want their servers set up exactly the same.

>Neither Ansible nor Puppet takes 6 months to configure. Yes you have to write the code (and/or pull pre-existing modules) to make the config changes you want, but that's the point of them: they are there to automate the changes of config on servers.

So if I've got 40 different models of servers, with 10 different models of switches, 8 different operating systems, you're telling me right now there's a playbook for every possible hardware configuration I could have that will update all of that to the latest firmware without significant work on my part? Where is this magical repo?

I think you misunderstand how Ansible is supposed to work. You can put in the work (much less than ~ 6 months, I'm available for freelancing!) and have it working. By then you will be able to run your command :-)

You will also be able to share as much of the deployment as you want/can between the different models/OSes. Templating configuration variables based on their characteristics.

What you're asking to is akin to "Why cannot I download an app that does what $PRODUCT does but for my business for free? Why do I need to write my own software?"

Free? I’m not sure if that’s a joke. Ansible licensing is about as far from free as you can get.

Having to hire someone to customize ansible is the exact opposite of what op asked for and just reaffirms my point: ansible does nothing without significant customization.

Ansible licensing? I must be missing something! Do you mean Ansible Tower? Ansible is free and OSS. It has AWX which is the Tower equivalent but free.

Yes. Ansible is a framework with plugins. Just like Django or Ruby on Rails.

What other ansible licensing would I be talking about? AWX has no support, it is a non-starter in the enterprise.

So again: Ansible is not an answer for someone looking for a ready-made solution to managing bare metal, which was my entire premise. Telling people to just use Ansible or Puppet when they're asking for a solution to the problem is just barely more helpful then telling them you can do it with a bash script and SSH for a consulting fee. It's pedantic and misses the point entirely.

I've used Ansible to manage bare metal. It worked great. I've used Ansible in high availability enterprise environments. Again, it worked great.

I'm not as big of a fan of Puppet but actually puppet also works great for on-prem systems given that's what it was originally designed for. If anything, Puppet makes less sense in the cloud than it does on bare metal.

I'm not disputing you have a complex problem but that just means you need to spend a little more time tuning your solution (not less time like you seem to assume).

And if you want my advice about how to approach a daunting build: break your problem down. First start with delivering easy systems which will have the least impact if it goes wrong. This is to get your confidence up in working with the tool. Then start picking the harder targets that will give you the most reward, so even if your project ends up unfinished you've still fixed the biggest problems in your org. Then work your way backwards until everything is fixed. After a while, some of the easier deliveries will become background jobs you can fit in between support queries or half day sprint tickets (depending on whether you Kanban or Sprint). Before you know it, you'll have everything automated and realise it was far less painful than it appeared before you'd started the project.

Disclaimer: I'm DevOps Manager who has transitioned several orgs to through this process :)

AWX (and Tower) have a different use case than the base apps. And I think you'd be surprised how heavy it is in the enterprise. You don't need to pay thousands to get battle-tested tools.

You don't need to use Tower to use Ansible.

You can use Ansible to manage bare metal - I've done it at two different companies.

Trust me, it works great.

> Having to hire someone to customize ansible is the exact opposite of what op asked for and just reaffirms my point: ansible does nothing without significant customization.

When you decide that your cattle is going to be pets and pets get individual care and feeding, you have to hire a lot of people to do the individual care and feeding.

I've done done management of ~2k Linux systems across 17 generations with Ansible. It is not a big deal. You enforce conventions so you no longer have 2k different servers but rather a feet of Betsys, a fleet of Franks, a dozen of Marshas and a couple of Jacks. And you do not touch the boot configuration because even in 2020 you do not need to touch the boot/network configuration.