Shell Script Best Practices, from a decade of scripting things

Re:Shell Script Best Practices, from a decade of scripting t... by usrbinbash on Thursday October 27, 2022 @10:13AM (#33357704)

> What is the Shell paradigm? I would argue that it's line-oriented pipelines.

Which python can do realitively well, by using the `subprocess` module.

Here is an example including a https://porkmail.org/era/unix/award (useless use of cat) finding all title lines in README.md and uppercasing them with `tr`

    import subprocess as sp
    cat = sp.Popen(
        ["cat", "README.md"],
        stdout=sp.PIPE,
    )
    grep = sp.Popen(
        ["grep", "#"],
        stdin=cat.stdout,
        stdout=sp.PIPE,
    )
    tr = sp.Popen(
        ["tr", "[:lower:]", "[:upper:]"],
        stdin=grep.stdout,
        stderr=sp.PIPE,
        stdout=sp.PIPE,
    )
    out, err = tr.communicate()
    print(out.decode("utf-8"), err.decode("utf-8"))

Is this more complicated than doing it in bash? Certainly. But on the other side of that coin its alot easier in python to do a complex regular expression (maybe depending on a command line argument) on one of those, using the result in an HTTP request via the `requests` module, packing the results into a digram rendered in PNG and sending it via email.

Yes, that is a convoluted example, but it illustrates the point I am trying to make. Everything outlined could probably done in a bash script, but I am pretty certain it would be much harder, and much more difficult to maintain, than doing this in python.

Bash is absolutely fine up to a point. And with enough effort, bash can do extremely complex things. But as soon as things get more complex than standard unix tools, I rather give up on the comfort of having specialiced syntax for pipes and filehandles, and write a few more lines handling those, if that means that I can do the more complex stuff easily using the rich module ecosystem of Python.

[ Parent | on HN ]

Re:Shell Script Best Practices, from a decade of scripting t... by lbhdc on Thursday October 27, 2022 @10:58AM (#33358523)

> But on the other side of that coin its alot easier in python to do a complex regular expression

I am not sure I would agree. Sed fills this role quite nicely.

cat README.md | grep # | tr '[:lower:] [:upper:]' | sed 's/something/something_else/'

[ Parent | on HN ]

Re:Shell Script Best Practices, from a decade of scripting t... by usrbinbash on Thursday October 27, 2022 @11:32AM (#33359112)

Now do that again, but this time the regular expression is controlled by 2 command line params, one which gives it the substitution, the other one is a boolean switch that tells it whether to ignore case. And the script has to give a good error if the substitution isn't a valid regular expression. It should also give me a helptext for its command line options if I ask it with `-h, --h`.

In python I can use `opt/argparse`, and use the error output from `re.compile` to do this.

Of course this is also possible in bash, but how easy is it to code in comparison, and how maintainable is the result?

[ Parent | on HN ]

Re:Shell Script Best Practices, from a decade of scripting t... by lbhdc on Thursday October 27, 2022 @12:00PM (#33359555)

In the example I gave I wouldn't write that in a script file, so I would just alter the command itself.

If I wanted to parse cli args I would use case on the input to mux out the args. I personally prefer writing cli interfaces this way (when using a scripting language).

    while test $# -gt 0; do  
      case "$1" in  
        -f|--flag) shift; FLAG="$1";;  
      esac  
      shift  
    done

[ Parent | on HN ]

Re:Shell Script Best Practices, from a decade of scripting t... by scbrg on Thursday October 27, 2022 @01:09PM (#33360704)

Man, you chose the wrong username, didn't you? ;-)

[ Parent | on HN ]

Re:Shell Script Best Practices, from a decade of scripting t... by usrbinbash on Friday October 28, 2022 @02:49AM (#33368404)

Not really, I love bash. I also love perl and vimscript btw. :D

[ Parent | on HN ]

Re:Shell Script Best Practices, from a decade of scripting t... by asicsp on Thursday October 27, 2022 @11:15AM (#33358803)

grep+tr can be done within sed too (or go with perl for more features and easier portability)

[ Parent | on HN ]

Re:Shell Script Best Practices, from a decade of scripting t... by sargstuff on Friday October 28, 2022 @11:40AM (#33372838)

one tool/command per 'concept' was a resource saving thing at one time.

sed is the thing that handles shell regular expressions for shellscripts.

[ Parent | on HN ]

Re:Shell Script Best Practices, from a decade of scripting t... by rascul on Thursday October 27, 2022 @01:05PM (#33360656)

> But on the other side of that coin its alot easier in python to do a complex regular expression (maybe depending on a command line argument) on one of those, using the result in an HTTP request via the `requests` module, packing the results into a digram rendered in PNG and sending it via email.

Doesn't sound so bad. A quick argument parser, a call out to grep or sed, pipe to curl, then to graphviz I guess (I don't really know much about image generation tools though), then compose the mail with a heredoc and run sendmail. Sounds like 10 to 15 lines for a quick and dirty solution.

[ Parent | on HN ]

Re:Shell Script Best Practices, from a decade of scripting t... by usrbinbash on Friday October 28, 2022 @02:08AM (#33368205)

It's certainly possible, but here comes the fun; How read/maintain/extend-able is the solution? How well does it handle errors, assist the user? Add checking if all the programs are installed and useful error messages into the mix. Then the API does a tiny change and now we need a `jq` between curl and graphviz, and maybe we'd need an option for that case as well, and so on, and so on, ...

Bash scripts have a nasty tendency to grow, sometimes in ways that are disproportional to the bit of extra functionality that is suddenly required. Very quickly, a small quick'n dirty solution can blow up to a compost-heap ... no less dirty, but now instead of a clean-wipe, I'd need a shovel to get through it.

I think my handle speaks for itself as to how much I like bash. But I have had the pleasure of getting handed over bash scripts, hundreds of lines long, with the error description being "it no longer works, could you have a look at it? and the original author both unreachable and apparently having string feelings against comments.

And in many of these cases, it took me less time to code a clean solution in Python or Go, than it took me to grok what the hell that script was actually doing.

[ Parent | on HN ]

Re:Shell Script Best Practices, from a decade of scripting t... by sargstuff on Friday October 28, 2022 @11:39AM (#33372822)

shell was originally tied to job/programm processing.

[ Parent | on HN ]