> You are source controlling your shell scripts right?

My personal heuristic for shell scripts are that if I care enough about them to put them under source control, they shouldn't be a shell script.

I have seen entirely too many Python scripts that are poor reimplementations of shell scripts that don't get the edge cases right, have more security vulnerabilities, etc. than a shell script a fifth of the size. There are things you simply can't do in other languages without excessive verbosity. (One that came up at my workplace recently is using <(...) to load a password from a secret store without writing it to disk or putting it in the environment.)

Use the right tool for the job, and then put it in version control (and add tests, too, by the way). Shell is the right language for tasks that primarily involve running lots of subprocesses, whether they are simple or complex. If a small part of your task needs functionality that can't be done well in shell, fortunately, shell is very good at running subprocesses, and it's a perfectly reasonable approach to do something like this:

    foo () {
        python3 -c 'import sys, foo; print(foo.bar(sys.argv[1:]))' "$@"
    }

    a="$(foo "$baz" "$quux")"

I regularly do this with the requests and json modules in particular, because being an HTTP client or a JSON parser is not a thing shell is good at. (For the specific problem of manipulating JSON, jq is another fine option if you have it installed.)

Beware that "python -c" prepends empty string (i.e., cwd) to sys.path, which is most likely not what you want in the context of a shell script.

> without excessive verbosity

If no-one (including you, unless you dream in bash) is ever going to modify or extend the script, OK. Otherwise python's readability/maintainability trumps excessive verbosity every time.

By "excessive verbosity" I do mean poor readability and poor maintainability. If you're invoking 5 commands and most of your Python code is wiring up the commands to each other right, just write 5 lines of shell, don't make me pull up the subprocess docs to see if your 30 lines of Python are doing the same thing and how to make a small change without risking pipes deadlocking.

Python is readable and maintainable when it's doing the things Python is suited towards doing. Running lots of external processes is not one of those. This is not a complaint or an insult to Python as a language (which I use regularly!), it is just a statement that different languages have different strengths and you should use the right tool for the job.

If you know of a good Python library that handles things like shell pipelines, <(...), and automatic creation of process groups (so that signal handling does the right thing), I'd be extremely interested, because I would like to use Python for these use cases. But it's currently the wrong thing for maintainable code for this one use case, and there is a very good language that handles this use case very well and is extremely stable and widely deployed.

I'd recommend checking out Plumbum (https://plumbum.readthedocs.io) -- at the very least, it has a solid base for easily setting up pipelines, input/output redirection, and signal handling.

Wow, thanks! I'll have to check this out.

Does the Plumbum library grant your wishes? I've not used it myself so I don't know, but its purpose is to make you "never write shell scripts again", so it might.

My personal experience would disagree, and I feel that is throwing caution to the wind.

As any software person knows, you really can't tell what's going to happen to the code/scripts you put out, not committing it to source control is a dangerous game to play.

If you have a simple shell script sitting on a server doing some basic task, why would you not have it under source control where it can be viewed by future teams and seeing what changes have been made to it over time? Seemingly simple problems can be caused by minor changes which are very visible if its under source control.

Just because its simple doesn't make it any less important. Complexity is not a good measure of its importance.

Especially when you start trying to implement IAC in legacy areas...

At that point, you care enough to put it under source control, and it shouldn't be a shell script. That's the entire point of the comment you replied to

And that's a stupid heuristic. Simple bash scripts can be critical to a processing pipeline while still being the best tool for the job.

It's that arbitrary distinction that I am responding to. I don't agree that you should reserve source control for the complex.

A shell script can be the appropriate tool for many important tasks.

I agree that simpler scripts deserve to be in source control too. I read it to mean what I also would say, myself: if it's anything but the most trivial of scripts (so, rather _programs_), they shouldn't be in BASH. Some people in this discussion are clearly very well versed in BASH. Great. For average developers, though, it's hard to build (good) programs in BASH, and the rabbit hole swallows them. Every time.

I completely agree, source control is not used to track complexity. It's to track changes and the reasons for those changes... however small they may be.

Straight from teenage wisdom on freenode ##linux circa 2003

Not sure I agree with this. We have lots of scripts that are shell scripts and are important enough to be in source control. I don't view shell scripts as throw away solely

My sequence is wiki steps, script, shebang (better scripting language), application. Every step is under version control of some form or other.

The shell script is just locking down some sequential command line interaction into something less error prone.

Do Makefiles fall under that category?

Chiming in for myself, no. Makefiles are more about dependency graphs and reproducible pipelines. I use them to make more sophisticated things than bash scripts, which are purely imperative.

I very much dislike cmake, and I've not yet been impressed by any of the other many Make replacement candidates.

How else would you accomplish what a Makefile would?

I think for C compilation, Tup is probably better than make. It also has a couple of small syntax improvements compared to make. In particular, it's much better with commands that loop over targets and dependencies.

However, as someone who doesn't write much C, I went back to make. Makefiles seem to adapt really well to different workflows.

A lot of my makefiles look like "download this publically available data, scp this secret data off a company server, mungle all the data with some scripts, shallow clone this branch of that repository, run the program in the repository a few times under different conditions, make a chart of the results".

Tup really doesn't like steps like "clone this repository". Make isn't exactly great with keeping track of targets that output entire directories, but it does work.

I also feel like, for me, make has become more powerful over time as I've learned other tools. Specifically: being able to just magic up any old environment using Nix and be confident that it won't break, mashing data with things like jq, cut, sed and awk, and quick database operations with SQLite.

I haven't used them enough to have an opinion because of questions like these, which I need time to answer for myself.

Honest question: what happens when the variable has a quote in it? If FOO is ‹xyz"; rm -r *; "xyz› (where I've used ‹› as delimiters) then won't even "${FOO}" expand to multiple arguments/commands? Or does bash automatically escape the quotes in the situation?

Quotes that arise from substitutions aren't considered to be quotes.

Unfortunately, the way GNU Bash handles this sort of requirement is to internally translate these protected quotes into some character that "nobody" would ever use, and then recover them later.

That characters code is none other than ASCII 1 (SOH/Ctrl-A). It's known by the preprocessor symbol CTLESC in the Bash sources.

jesus christ is nothing sacred

Both of those examples end up being fine. When quoted the entire variable is passed as a single WORD and the shell won't interpret the `;` as the end of the command. Even without quoting the variable it would be fine in this case because of where you've put the quotes, though you should still quote the variables. As a simple illustration take a look at the following:

    function e() {
        echo $#
    }
    $ x="hello world"
    $ e $x
    2
    $ e "$x"
    1

I agree. Consistency wins over typing 2 extra characters. It's much easier (for me at least) to scan shell scripts and notice the variables when they are written consistently.. and that means using braces.

When I saw that recommendation it made be sad. I love seeing my variable names with a quick glance. Came to the comments mostly to see if anyone had the same preference.

Can't help but this brings to mind a recent article on medium about about "The irrational love for curly braces" (in the context of web frontend architecture and markup languages but still).

[1]: https://medium.com/@fagnerbrack/front-end-separation-and-the...

> [1]: "${foo} bar baz" v.s. "$foo"" bar baz"

Why would you quote the second option like that? You can just write: "$foo bar baz"

I don't think your example backs up your point about it being easier on the eyes. This would also work:

    "$foo bar baz"

Since there's a space after foo, the braces don't actually do anything and thus _can_ be eliminated. Your other points make a decent argument for _not_ eliminating them even though you can, but removing extra characters makes it much easier to read.