Ha, Ansible parses and 'executes' YAML files using Python, but not before passing them (repeatedly?) through a template library (Jinja2) originally designed to produce HTML pages. They even came up with a way to write for loops in YAML... What I'm trying to say is that sometimes great software is written using (in some people's opinion) terrible technology. Not everything needs to be written in idiomatic Haskell or Rust.

Personally I have to say that I started writing more bash scripts again recently, I even wrote one that interfaces with the GPIO ports on a Raspberry Pi. Works like a charm, is concise and (I find) also quite readable: https://gist.github.com/adewes/7a4c20a5a7379e19d78ba54521d3d...

But then again I'm also a person who enjoys Perl, so maybe it is time for me to be banished to a lonely island :D

> originally designed to produce HTML pages

Jinja2 can produce HTML pages, but it was never exclusively designed for that. In fact one of the earliest uses of Jinja2 was generating LaTeX.

I stand corrected, didn't know that!

Perl was great for encouraging that kind of thinking with the old motto "there's more than one way to do it". I'm glad I started with Perl because it's helped me think about solutions I may have otherwise dismissed.

Bash is far from perfect, but it's readily available at many people's fingertips, and it gets you to some solutions really fast. This is cool.

Yes it is.

If you have two type systems and you don't want them to interact, gluing them together with stringly typed bash is a good way to keep their scope small. As a user interacting with multiple programs by different authors, that's a very common use case.

Giving your type systems boundaries is a good idea. The alternative tends towards one-type-system-to-rule-them-all, an idea that Russell thoroughly explored in the early 1900's, Godel proved to be unmaintainable in the 30's, and that everybody except the Java and C# crowd has since abandoned as a dead end.

It is a pretty sloppy and archaic language. We need an updated version that keeps the string typing and inline execution of external binaries, but has modern syntax, plus a few more conveniences and guardrails.

You might be interested in Powershell. I used bash for 20 years and considered myself an experienced user and programmer with it. Comparatively, my experience with Powershell is relatively short, but I switched to it and the first time I needed to handle a piece of structured data I was hooked and haven't looked back.

I recently discovered Powershell, in a Windows context. It's weird, but has interesting ideas, and is better than bash for sure.

Supposedly it supports Linux, but I haven't actually tried.

It does, and MacOS, and I found that once I changed to bash-style tab-completion and used PSReadLine to get the keybindings I was familiar with, it becomes a pretty comfortable shell. I remove some default shell aliases because PowerShell's notion of file modes and permissions ends up not mapping well, but you have all your normal Unix commands available so it's not really a big deal. Quoting is weird but it's not worse than bash, where quoting is also weird. It's surprising how well objects-as-arguments and objects-as-pipelines work, even when mixing with standard Unix commands. I appreciate standard parameter handling, command naming and I think it pays off on the "Unix philosophy" (small program units doing one thing well) better than bash and shell tools. Those standard approaches produce an aesthetic I didn't like for a long time (Invoke-RestMethod -Url ...) but aliases cut down on your typing (irm instead of Invoke-RestMethod) and I now appreciate the discoverability of standard commands. I keep discovering little gems like default parameter values and the various ConvertFrom-* commands, some from modules.

I was only taking issue with the objection to string typing. I totally agree that the rest of bash requires a bit of stockholm syndrome to love.

I haven't yet taken the time to play with it but I'm hoping www.oilshell.org addresses some of the warts.

I don't disagree with your point, in this context. Bash can be good enough for all kinds of things. Or maybe not. But it's not a simple binary equation--there are tradeoffs that might lead you to it or away from it for a particular situation.

I do think there are a few issues that make bash non-robust for this kind of task. If you only look at this through the lens of the "type system", you have a good point about gluing views together, but I think "just strings" is not enough typing. I think I and the industry have had better luck with a base system more like JSON, instead of "just strings". It provides enough structure so that there's a straightforward way to turn wire protocols or other external data into your Java objects or whatever, let dynamic scripting languages handle it in their native base types (which often map pretty well to what JSON offers, while bash really doesn't--note that the author here isn') and are fairly reasonable to manage on their own without special tooling.

However, that aside, even if "just strings" were fine for everything, the fact that bash is designed as an interactive shell with barewords for strings really exposes a lot of gotchas for handling data from different sources, just from the sometimes-diabolical problems with quoting and I/O. In the absence of actual function return values, capturing output and storing a process exit code, which might sometimes be the last process's exit or maybe something inside the pipeline if you've set the right global options, is error-prone. It's not robust to hope that the only output coming (on two different output streams) constitutes a complete and correctly-anticipated return value from a subprogram. Errors associated with empty variable values are frequent for this reason.

Secondly, bash lacks a lot of features which you need external programs to provide, and the dependency management glue between them is not good. You'll need find, ls, sed, awk (and you may not remember the days with multiple significant Unix platforms and differences in behavior between gawk, nawk, awk-of-different-flavors), cut, sort, ps (/usr/bin/ps or /usr/ucb/ps) all stored in different platform- and environment-specific locations and exhibiting different behaviors without a clear way of constraining your dependencies or even asserting them correctly. In the modern world, you can add to that list curl, jq, parallel, and other shell tools. Fairly common tasks like YAML handling is even more of a clusterf. The fact that these tools were all designed to work for their own reasons and not as extensions to a common language isn't their* problem, but it becomes yours if you're trying to deliver a robust and serious piece of software with it. You'll do a lot of DIY around discovering their edge cases and handling them.

There are a number of other problems but I think they're lesser in scale, like the lack of a standard way to deliver and use extensions. It's hard to do safe file access. Bats is a pretty good testing framework but no one (including the author of this script) seems to use it. But what I don't really see are many advantages over a regular scripting language, for something like this. The fact that "bash is already installed" is only an advantage as long as your script isn't going to do anything interesting; once it starts doing interesting things, it needs to worry about all kinds of other things being installed.

That said, I do think it's a little unfair to tear the author apart over something that really shouldn't be any more ambitious than a cron job doing "git clone https://config-repo/config.git && config/manage.sh". The fact that they call it an "alternative to Ansible" is a little eyebrow-raising, but I'm sure it is: to them. The question isn't really whether it does what Ansible does, but if you can manage a machine's configuration with a shell script, and of course the answer is yes. But the answer to the question, "Is bash a fine language to write a general-purpose configuration management system in?": the answer is no.

I don't really mind the string typing of Bash. Shell commands, their arguments, and their outputs are all strings anyway so it makes sense to give special treatment for strings.

That said, features such as word-splitting are terrible footguns that only exist because of the burden of backwards compatibility. Hackernews, what other languages do you suggest as Bash replacements that doesn't have as many opportunities to shoot yourself on the foot?

This is an area that I never found something I truly liked. A lot of people suggest Python for this kind of task but when I have a small script that is mostly doing pipelines and command substitutions with "$()" then the python version with subprocess is much more verbose.

There's a wonderful library called `plumbum` [0] that lets you execute external binaries and capture output using Bash-like piping syntax, while preserving all the goodness of Python's syntax and flow control. Highly recommended.

[0] https://plumbum.readthedocs.io/en/latest/

Are you really really really doing mostly pipelines and command substitutions? Most of the time when opening a script with such claim there is maybe one pipe that makes sense and all the others are just things like cut, grep and awk to trim of any trailing newlines. Substitutions are usually from things like date. All of things that are easier to manipulate in a typed language.

Can't say anything about this tool, but compared to ansible, bash is mostly fine. In my opinion ansible is a (and pardon my language) clusterf* of a tool compared to the alternatives. Certainly, if bash had the likes of inventories, roles and other features to make maintenance and reuse easier, I'd choose that over ansible.

Ansible is writing bash scripts in YAML, templates using system that doesn't grok YAML and causes parsing errors due to whitespace, interpreted by schizophrenic Python that might be 2 or it might be 3.

... I guess pure bash might be better.

I have my issues with Ansible, and I'm really not a fan of "templated YAML"-based tools, which has been an unfortunate trend that's not limited to Ansible (GCDM and helm share this problem). But if you're writing a lot of bash scripts into your plays you're probably doing it wrong.

you can run pure bash from ansible, either by copying a script or using the shell module: examples: both with sudo

ansible all -m shell -a "hostname" -b

ansible all -m script -a "script=/home/user/myscript.sh" -b

I love to throw together a quick inventory list and run adhoc ansible commands against a bunch of servers.

When I open `av` in vim, shellcheck spits out 167 warnings. This is not written robustly.

FWIW shellcheck complains about many false positives.

But in this case, their code is comparing strings to ints like on line 126:

    while [[ "$#" -gt 0 ]]; do

Spot checking code for basic bash mistakes like this tends to tell me the quality of the code overall, as these are basic mistakes. $0.02

If $# is a number, aren’t "$#" and $# exactly the same in Bash? Enclosing things in double quotes doesn’t change their type, since everything is already treated as a string.

https://www.gnu.org/software/bash/manual/html_node/Double-Qu...

While the other reply to you is correct, the actual answer to your question is No -- bash interprets leading 0s (zeros) in the C sense, it indicates octal not decimal. So this very small bash will fail for you:

    X="08"
    if [[ "$X" -eq 8 ]]; then
      echo Y
    fi

Use `bash -x` to run it in debug mode:

    $ bash -x test.sh 
    + X=08
    + [[ 08 -eq 8 ]]
    test.sh: line 2: [[: 08: value too great for base (error token is "08")

This comes into play if you're dealing with things like IPv4 address, where it is technically allowable to have leading zeros, however by convention it's never done because of code traps which see those leading zeros as octal mode, not decimal.

Your example only demonstrates the effect of `[[`. If you replace it with `[` the error won't occur. In both cases double-quoting (or not) `$X` makes no difference in the context of your example.

[ is an external program called "test" and [[ is a bash builtin, they have many different behaviours beyond just trying to use one to prove the other wrong on a random internet comment.

I merely used that to show that the error message was tied to `[[` and not to the use of double quotes (which your parent comment was asking about).

It's true, but you're semantically overloading the variable syntax (by hinting that it's treated as string).

Plus, left and single operands, in bash conditions, don't require quoting, so a lean style is actually not to use them at all.

> …left and single operands…

That's false:

  x='foo bar'; [ $x = 'foo bar' ]; echo $?
  -bash: [: too many arguments

  x='foo bar'; [ "$x" = 'foo bar' ]; echo $?
  0

As I replied to your other comment, you cannot just replace [[ with [ and get the same behaviour. If you use the actual bash builtin [[ which we're discussing, it works as expected. Also, please use == when comparing, = is an assignment operator (in almost every programming language).

My bad. I interpreted “bash conditions” differently.

Is this actually a shellcheck error? It just evaluates to the integer number of arguments in $@, so I doubt it is (and I doubt it matters whether it's quoted? unless there's a bash back-compat issue here?)

Is there a bug in the command line parsing there? I can't see it.

In this specific case it's more of a nitpick, I think. Even shellcheck won't demand double-quoting `$#` because everyone(?) knows that it'll always be a number and have no whitespace in it nor evaluate to an empty string.

Do you refer to shellcheck in general or in this specific case?

I do a non-trivial amount of shell scripting, and there is only one warning I disable in shellcheck on a general basis (SC2016).

The author of this script clearly doesn't like quoting where the underlying value is guaranteed not to include special characters (or space), although this is somewhat inconsistent with the fact that he's using quotes in several patterns where it's not needed (eg. left or single operands in bash conditionals, or in variables initialization).

On can consider always quoting a defensive practice. In a fragile language like Bash, I personally think defensive practices are very important, so the first impression I get from a Bash program not adopting them is "not great".

It whines at `ldd`, too, but that ain't stopping me.

I think you are forgetting that serious operators do not have spaces in their paths.

*People who are used to work with bad tools don’t have spaces in their paths.

"C:\Documents and Settings" still hurts after all these years.

It still hurts today, because Microsoft puts a space between your first and last name in your profile if you are using a local account but not for a domain account, which causes all kinds of weird issues if you ever have to use cmd.exe...

"C:\Program Files(x86)". Who ok'd that?

It's a design decision and a good one, at that. Forcing 1 million developers to handle spaces properly while they spend 1 billion hours developing their software is preferable to 1 billion regular users having to remember to not use space, a perfectly good separator, in their 1 billion billions hours of usage of said software.

Speculation, but I suspect that Microsoft explicitly put both user data and programs in paths with long filenames and spaces to help force software to support those things. That always seemed like a good thing to me, it's ridiculous that we still have software that breaks on paths with spaces.

I haven't looked in a while, but last I checked you could also still get to it via C:\PROGRA~1? (This was years ago, that might have been 'fixed'?)

These are called 8.3 filenames. They're a filesystem feature. They're still created by default in Windows 10 on NTFS, but can be disabled or stripped using `fsutil`.

cough make cough

I don't know about "serious," but trustworthy operators use tools that work correctly in the presence of weird states because they take responsibility for their work instead of blaming others.

"Serious operators" never want to put their files into iCloud Drive? They must be using butterflies then

https://xkcd.com/378/

"Serious operators" probably use a cloud storage solution that supports WebDAV/FTP/SFTP, like box.com or Google Drive - "real operators", on the other hand, have a self-hosted solution (-; And IMHO relying on iCloud for essential data storage probably isn't a good idea.

I find iCloud much better than Google Cloud and I'm pretty sure I qualify as a "serious/real operator" any way you want to call it. I have no time or interest in hosting a custom solution. And I certainly have no need for FTP.

But even if I wasn't experienced, quit this gatekeeping crap. Let people use the tools that work for them. It does not in any way reflect on how "serious/real" they are.

> "Serious operators" probably use a cloud storage solution that supports WebDAV/FTP/SFTP, like box.com or Google Drive - "real operators", on the other hand, have a self-hosted solution (-;

I didn't mean to discredit you for using iCloud - if it came across that way, I apologize. My original comment was meant to be tongue in cheek. I've got to admit that I'm a bit gob-smacked by your reaction... After all, my comment was similar in structure to yours. I even quote unquoted the terms, and the "real operator" part was a reference to the editor war xkcd you mentioned. Taking your post literally, I could accuse you of doing the same thing by assuming that real operators were Mac users. And I'm sure there are many use cases where hosting your own cloud storage would be absurd. But, re-reading my comment, it didn't read like a flame war instigating piece of propaganda "Calling all real sysops to arms against the user threat. Grab your cattle prod, fellow comrades, and crank up the voltage!"

> And IMHO relying on iCloud for essential data storage probably isn't a good idea.

I don't condone gatekeeping (although, to be perfectly honest, I don't see how my comment could be interpreted as elitist. It's not like the learning curve for GDrive or any other solution with a better cross platform support is _that_ high), but I don't see a reason why I shouldn't voice my opinion on a tool, even if responding to something obviously meant as a joke. Of course you're free to use iCloud. Personally, I wouldn't feel comfortable with the thought of my critical data being stored on a platform that, historically, hasn't always been known for being extremely stable. Maybe I'm just biased. If you feel comfortable using it, then use it. I won't stand in your way.

I genuinely apologize as I think I misunderstood. Of course I'm fine with a bit of jokes about tech but I think I didn't catch any intended sarcasm in your comment—in fact I think I took the direct opposite intention.

I'm admittedly sensitive to gatekeeping. I do think it's one of the biggest reasons the tech industry is the monoculture it is.

Anyhow, I appreciate you clarifying and hope I didn't offend too much.

No problem. The subtleties of human speech don't carry over to text very well ;-)

rsync or btrfsync for the win

On the contrary, serious operators write programs to handle spaces in the paths.

That’s just objectively false.

> No, it's not.

Then come up with something better.

It will need the same or better ease-of-use for non-programmers, should function as a shell (so that their everyday use also teaches them scripting), with the same flexibility that lets people who don't care about typing or other fancy programmer stuff and just need to get work done.

If your first instinct is to say "python", you don't understand the problem.

A Python script is (generally) easier to read, easier to write, easier to modify/maintain, easier to debug and easier to test.

Bash programming is not easy and full of loads of gotchas.

Useful for running multiple commands but when you start to need to do programming things like conditionals/loops/variables then using an actual programming language makes more sense.

You are misunderstanding or misrepresenting what I said. If it wasn't obvious, I meant that bash scripting is not okay _for this sort of complex work/projects_.

On paper I agree with you, but that has not been our experience in practice.

Bash "just works" with some very well understood limitations. Every one of these "better" solutions introduces 10x the complexity with 10x the limitations (always undocumented).

After spending dozens of hours configuring them thanks to terrible documentation, generally taking multiple support tickets (if there's anyone to talk to at all), we get it working. Then, after using it for a period of time, they fall flat on their face in far worse ways than bash.

Granted I'm only managing hundreds of servers, not thousands or more, but more often than not a

    for i in {000..199}; do ssh machine$i "some command"; done

is more reliable, with better error handling, than all of the other solutions.

Edit: As long as linux/unix maintains the "everything is a (plaintext) file" approach, I have trouble picturing a better solution than text/string based interface like Bash. You can build more front-end complexity on top of it, but you're always going to end up back in the same place.

> Granted I'm only managing hundreds of servers, not thousands or more, but more often than not a

> for i in {000..199}; do ssh machine$i "some command"; done is more reliable, with better error handling, than all of the other solutions.

This is absolutely not the case:

From the top of my head: there is a network error that hits while running this against 5% of machines. What is the state of those servers?

If one is managing more than a few machines, one should absolutely use something like Puppet/Ansible/Salt/Chef with a repo to ensure that nothing is ever lost.

And all of those tools (at least the ones I've looked at) use SSH internally as well and are just as susceptible to network errors, except they tend to fail with less feedback than the raw ssh command.

I never take it on blind faith either:

for i in {000..199}; do ssh machine$i "echo new_setting > /file"; done

for i in {000..199}; do ssh machine$i "grep new_setting /file"; done

for i in {000..199}; do ssh machine$i "reload service"; done

for i in {000..199}; do ssh machine$i "verify reload"; done

As I was taught "never make a change without verifying it". All changes are documented as well.

> SSH internally as well and are just as susceptible to network errors, except they tend to fail with less feedback than the raw ssh command.

Maybe some. Puppet, for example, fetches a manifest and applies it on the server. Failure to apply a manifest triggers re-running the manifest. In small installations it will be asking puppetmater(s). Large installation run masterless, where they fetch the repo and run it. I would be highly surprised if Ansible did not have this mode even if it is not a popular way of doing it.

> for i in {000..199}; do ssh machine$i "echo new_setting > /file"; done > for i in {000..199}; do ssh machine$i "grep new_setting > /file"; done > for i in {000..199}; do ssh machine$i "reload service"; done > for i in {000..199}; do ssh machine$i "verify reload"; done

You are running this sequentially and by hand?! What happens when the server is dead and you need a new one? You provision it by hand and go through the stages to get it into the state where it can accept the same cattle command?

That's fair. Maybe it's worth looking into again.

> You are running this sequentially and by hand?! What happens when the server is dead and you need a new one? You provision it by hand and go through the stages to get it into the state where it can accept the same cattle command?

Depends on the change, but generally something like that. Change + Verify. It really doesn't take much more than 10-15 minutes for any given change. We've been at a stable state for about 2 years so changes are rare at this point, other than our security update cycle.

For rebuilds, we pull periodic disk images (monthly + any major changes) and can stand up a new server in about 20 minutes, almost all of which is hands-off.

We also use base image which is a core OS configured to our liking with puppet and minimum bootstrap core that allows the system to ask "What am I supposed to be, again?" If we know that any of the instances are currently functioning or were functioning as intended than we know that new instance will be functioning as intended.

After that it takes between 60 seconds and 5 minutes for it to be inserted into service regardless of the complexity of its configuration and registration.

I have the complete opposite experience.

Places that have hundreds of bash scripts that they manually run in a loop like your example are normally a one off nightmare of random changes in random places with zero consistency, tons of edge cases, and a lot of failure scenarios.

Ansible is very entry level friendly, we've had devs grok it in 20-30 minutes and write their own playbooks. We've administrated thousands of servers at scale. We can even use it to package immutable servers with Packer and Terraform, with mostly expected results.

I'd choose the latter every time.

In the end of the day it's a collection of python modules that people have invested in that let you write steps as plainly as possible in yaml with some light logic involved.

Why reinvent the wheel?

We do realize that's a downside and are very careful about documentation. We only have 2 admins on the system, and both of us are well aware of everything that has been done.

Our experience was Ansible was amazing until it isn't. It worked for a while, then any failures would leave systems in an unknown state that had to be verified via adhoc testing like this anyway.

Then, when Ansible ran into errors, it was just a pain to get working again. Granted we haven't tried in a couple years.

Funny, because to us, Ansible just feels like reinventing the wheel. It's a more complicated tool for pushing text based changes to a group of servers.

I fail how to see how I could be doing either.

"[W]e need to stop it with the stringly-typed, footgun-filled madness that is bash scripting" is pretty categorical.

While I agree with the sentiment that bash is not the solution for an ansible-like thing, I also agree with what you're saying.

I consider myself an accomplished system engineer and software developer. I use bash all the time. Not writing bash scripts is like saying not to use the terminal, it's just cacophony.

With bash at least I don't have to read the documentation each time I want to create a file or anything else...

> We deserve better than this.

What have we done to deserve better? Have you seen a community of people working on a bash scripting replacement lately? Has it hit HN? Does it have the same confluence of features, convenience, portability, simplicity, and utility that bash scripting has? If so, why aren't we using it yet?

The fact is, we all use bash because it's the best we have for what we really want to do. Bash is what we deserve. 31 years of "good enough".

> No, it's not. It's 2020, we need to stop it with the stringly-typed, footgun-filled madness that is bash scripting.

Ansible is written in Python. You could make the same argument.

How? Python has an actual type system. A programming language that does strings for everything would be, for example, PHP.

Python has dynamic typing, which can be equally dangerous. In fact I’d argue it’s more dangerous. You will not throw errors until runtime, which will literally blow up in your face.

> which can be equally dangerous

Bash is weakly typed, which is a different concept than dynamic typing.

Dynamically typed languages prevent bugs that weakly typed languages don't, so from this point of view, the latter are unambiguously "more dangerous" than the former.

A dynamic type system is an incremental improvement over a weakly typed type system, and still will not guard against runtime errors. They are equally dangerous w.r.t runtime errors.

Not that I endorse the “equally dangerous” comment, but if you're interested in configuration management with a language based around a powerful static type system, may I note the existence of Propellor? https://propellor.branchable.com/

The concept of "guard against runtime errors" doesn't make much sense.

Technically speaking, even languages with strong safeguards (random examples: Golang, Rust) can't "guard against runtime errors", because it's possible to panic a program by performing an out-of-bounds array access (which classifies as "runtime error" in a very strict sense).

A concept that makes sense is the amount of safeguards provided by a language (/type system), however, this depends entirely on the use case; it's an intentional choice of the developer.

It gives you the option to use static typing, if you so desire.

I'm not sure how not having a type system is safer than having a type system. What makes it feel more dangerous?

When talking about Python in 2020 i think most people imply you are running Mypy, which does a very good job at statically analyzing python almost to strongly typed language level. Plus, as others said, weak and dynamic are not same thing.

So does Bash, though.

And Bash does not support static typing at all, unlike Python. Not sure what their argument even is.

Python doesn't support static typing either unless you're using mypy which is experimental. Bash and Python both however have strong types, although Bash's are weaker than Python's.

Right, but Python gives you the illusion of safety. And the person I was replying to was trying to dismiss this interesting Ansible alternative because it was written in a language he doesn’t like.

Python with type annotations can actually be pretty safe. There are also typechecked Python variants being developed. At any rate, I’d say the defect rate in Bash is incredibly higher than it is in Python, and 90% of that is due to unquoted string expansion. Try putting spaces in your paths and you’ll quickly see how 75% of the Unix stack breaks.

Somehow that is not a real problem in practice. Maybe whitespace is not needed in paths that much. If a need to store a random byte string in a file name appears once in a year, one can encode it into hexadecimal or base64 with % replacing /.

That's sort of like saying, lack of pointer safety is not a real problem in practice. Yes, you can work around it, but when it's a problem for production systems dealing with sensitive or untrusted data it's a huge problem.

I don't see how the two compare. As a programmer, you need pointers for dynamic allocation. You don't need to use whitespace in paths, ever. It is a convenience for the user.

Whitespace in paths occur, similar to how buffer overflows occur. It's the job of the programmer to understand this and guard against it; but many don't. And all you need is one bash script that does something like this

  rm -rf $path

to go from working one day to having deleted an entire directory tree just because someone named their file

  "foo ."