Safe ways to do things in bash

Re:Safe ways to do things in bash by geofft on Monday May 14, 2018 @11:12PM (#17071381)

I have seen entirely too many Python scripts that are poor reimplementations of shell scripts that don't get the edge cases right, have more security vulnerabilities, etc. than a shell script a fifth of the size. There are things you simply can't do in other languages without excessive verbosity. (One that came up at my workplace recently is using <(...) to load a password from a secret store without writing it to disk or putting it in the environment.)

Use the right tool for the job, and then put it in version control (and add tests, too, by the way). Shell is the right language for tasks that primarily involve running lots of subprocesses, whether they are simple or complex. If a small part of your task needs functionality that can't be done well in shell, fortunately, shell is very good at running subprocesses, and it's a perfectly reasonable approach to do something like this:

    foo () {
        python3 -c 'import sys, foo; print(foo.bar(sys.argv[1:]))' "$@"
    }

    a="$(foo "$baz" "$quux")"

I regularly do this with the requests and json modules in particular, because being an HTTP client or a JSON parser is not a thing shell is good at. (For the specific problem of manipulating JSON, jq is another fine option if you have it installed.)

[ Parent | on HN ]

Re:Safe ways to do things in bash by jwilk on Tuesday May 15, 2018 @04:09AM (#17072393)

Beware that "python -c" prepends empty string (i.e., cwd) to sys.path, which is most likely not what you want in the context of a shell script.

[ Parent | on HN ]

Re:Safe ways to do things in bash by foxylad on Tuesday May 15, 2018 @12:24AM (#17071634)

> without excessive verbosity

If no-one (including you, unless you dream in bash) is ever going to modify or extend the script, OK. Otherwise python's readability/maintainability trumps excessive verbosity every time.

[ Parent | on HN ]

Re:Safe ways to do things in bash by geofft on Tuesday May 15, 2018 @12:31AM (#17071667)

By "excessive verbosity" I do mean poor readability and poor maintainability. If you're invoking 5 commands and most of your Python code is wiring up the commands to each other right, just write 5 lines of shell, don't make me pull up the subprocess docs to see if your 30 lines of Python are doing the same thing and how to make a small change without risking pipes deadlocking.

Python is readable and maintainable when it's doing the things Python is suited towards doing. Running lots of external processes is not one of those. This is not a complaint or an insult to Python as a language (which I use regularly!), it is just a statement that different languages have different strengths and you should use the right tool for the job.

If you know of a good Python library that handles things like shell pipelines, <(...), and automatic creation of process groups (so that signal handling does the right thing), I'd be extremely interested, because I would like to use Python for these use cases. But it's currently the wrong thing for maintainable code for this one use case, and there is a very good language that handles this use case very well and is extremely stable and widely deployed.

[ Parent | on HN ]

Re:Safe ways to do things in bash by coandco on Tuesday May 15, 2018 @02:47AM (#17072108)

I'd recommend checking out Plumbum (https://plumbum.readthedocs.io) -- at the very least, it has a solid base for easily setting up pipelines, input/output redirection, and signal handling.

[ Parent | on HN ]

Re:Safe ways to do things in bash by geofft on Tuesday May 15, 2018 @08:51AM (#17073536)

Wow, thanks! I'll have to check this out.

[ Parent | on HN ]

Re:Safe ways to do things in bash by swinglock on Tuesday May 15, 2018 @12:16PM (#17075082)

Does the Plumbum library grant your wishes? I've not used it myself so I don't know, but its purpose is to make you "never write shell scripts again", so it might.

[ Parent | on HN ]