I have seen entirely too many Python scripts that are poor reimplementations of shell scripts that don't get the edge cases right, have more security vulnerabilities, etc. than a shell script a fifth of the size. There are things you simply can't do in other languages without excessive verbosity. (One that came up at my workplace recently is using <(...) to load a password from a secret store without writing it to disk or putting it in the environment.)

Use the right tool for the job, and then put it in version control (and add tests, too, by the way). Shell is the right language for tasks that primarily involve running lots of subprocesses, whether they are simple or complex. If a small part of your task needs functionality that can't be done well in shell, fortunately, shell is very good at running subprocesses, and it's a perfectly reasonable approach to do something like this:

    foo () {
        python3 -c 'import sys, foo; print(foo.bar(sys.argv[1:]))' "$@"
    }

    a="$(foo "$baz" "$quux")"

I regularly do this with the requests and json modules in particular, because being an HTTP client or a JSON parser is not a thing shell is good at. (For the specific problem of manipulating JSON, jq is another fine option if you have it installed.)

Beware that "python -c" prepends empty string (i.e., cwd) to sys.path, which is most likely not what you want in the context of a shell script.

> without excessive verbosity

If no-one (including you, unless you dream in bash) is ever going to modify or extend the script, OK. Otherwise python's readability/maintainability trumps excessive verbosity every time.

By "excessive verbosity" I do mean poor readability and poor maintainability. If you're invoking 5 commands and most of your Python code is wiring up the commands to each other right, just write 5 lines of shell, don't make me pull up the subprocess docs to see if your 30 lines of Python are doing the same thing and how to make a small change without risking pipes deadlocking.

Python is readable and maintainable when it's doing the things Python is suited towards doing. Running lots of external processes is not one of those. This is not a complaint or an insult to Python as a language (which I use regularly!), it is just a statement that different languages have different strengths and you should use the right tool for the job.

If you know of a good Python library that handles things like shell pipelines, <(...), and automatic creation of process groups (so that signal handling does the right thing), I'd be extremely interested, because I would like to use Python for these use cases. But it's currently the wrong thing for maintainable code for this one use case, and there is a very good language that handles this use case very well and is extremely stable and widely deployed.

I'd recommend checking out Plumbum (https://plumbum.readthedocs.io) -- at the very least, it has a solid base for easily setting up pipelines, input/output redirection, and signal handling.

Wow, thanks! I'll have to check this out.

Does the Plumbum library grant your wishes? I've not used it myself so I don't know, but its purpose is to make you "never write shell scripts again", so it might.

My personal experience would disagree, and I feel that is throwing caution to the wind.

As any software person knows, you really can't tell what's going to happen to the code/scripts you put out, not committing it to source control is a dangerous game to play.

If you have a simple shell script sitting on a server doing some basic task, why would you not have it under source control where it can be viewed by future teams and seeing what changes have been made to it over time? Seemingly simple problems can be caused by minor changes which are very visible if its under source control.

Just because its simple doesn't make it any less important. Complexity is not a good measure of its importance.

Especially when you start trying to implement IAC in legacy areas...

At that point, you care enough to put it under source control, and it shouldn't be a shell script. That's the entire point of the comment you replied to

And that's a stupid heuristic. Simple bash scripts can be critical to a processing pipeline while still being the best tool for the job.

It's that arbitrary distinction that I am responding to. I don't agree that you should reserve source control for the complex.

A shell script can be the appropriate tool for many important tasks.

I agree that simpler scripts deserve to be in source control too. I read it to mean what I also would say, myself: if it's anything but the most trivial of scripts (so, rather _programs_), they shouldn't be in BASH. Some people in this discussion are clearly very well versed in BASH. Great. For average developers, though, it's hard to build (good) programs in BASH, and the rabbit hole swallows them. Every time.

I completely agree, source control is not used to track complexity. It's to track changes and the reasons for those changes... however small they may be.

Straight from teenage wisdom on freenode ##linux circa 2003

Not sure I agree with this. We have lots of scripts that are shell scripts and are important enough to be in source control. I don't view shell scripts as throw away solely

My sequence is wiki steps, script, shebang (better scripting language), application. Every step is under version control of some form or other.

The shell script is just locking down some sequential command line interaction into something less error prone.

Do Makefiles fall under that category?

Chiming in for myself, no. Makefiles are more about dependency graphs and reproducible pipelines. I use them to make more sophisticated things than bash scripts, which are purely imperative.

I very much dislike cmake, and I've not yet been impressed by any of the other many Make replacement candidates.

How else would you accomplish what a Makefile would?

I think for C compilation, Tup is probably better than make. It also has a couple of small syntax improvements compared to make. In particular, it's much better with commands that loop over targets and dependencies.

However, as someone who doesn't write much C, I went back to make. Makefiles seem to adapt really well to different workflows.

A lot of my makefiles look like "download this publically available data, scp this secret data off a company server, mungle all the data with some scripts, shallow clone this branch of that repository, run the program in the repository a few times under different conditions, make a chart of the results".

Tup really doesn't like steps like "clone this repository". Make isn't exactly great with keeping track of targets that output entire directories, but it does work.

I also feel like, for me, make has become more powerful over time as I've learned other tools. Specifically: being able to just magic up any old environment using Nix and be confident that it won't break, mashing data with things like jq, cut, sed and awk, and quick database operations with SQLite.

I haven't used them enough to have an opinion because of questions like these, which I need time to answer for myself.