News for hackers. Stuff that matters.
Re:When Bash scripts bite by arghwhat on Friday May 12, 2017 @04:43AM (#14322341)
I'd much rather have 200 lines of bash than the 300-400 lines of python to replace it.

I have recently rewritten a lot of Python and Perl (over 10kLOC) to a small collectiom of 10-ish bash scripts that weigh in at about 700LOC total. There are some tasks where Bash is the only appropriate tool.

Much simpler code, much easier to read, and a lot faster too.

Re:When Bash scripts bite by growse on Friday May 12, 2017 @04:59AM (#14322406)
If you're measuring code quality / maintainability by how many lines of code there are, you're doing it wrong. I'd much rather see 400 lines of well-structured python than 200 lines of tangled bash. For the rewrite you did, how much of the benefit was because it was rewritten, rather than being rewritten _in bash_?

I'd also go as far as saying that if you're using bash because it's faster than python, you're also using the wrong tool for the job.

Re:When Bash scripts bite by arghwhat on Friday May 12, 2017 @05:38AM (#14322558)
The quality of the bash far outshined the quality of the python and perl, because Python and Perl was being used to do a bash scripts job. When what you need is to call a bunch of binaries and redirect output around, then Python and Perl just don't cut it.

The code is much, much easier to read now. You can see a script on a single screen height, which works wonders for overview. The 3x performance gain spawned from enlightenment this overview granted us.

But don't get me wrong, in the very same rewrite, I rewrote 100 lines of bash + 300 lines of Python into a single 200 lines of Python, which when combined with PyPy, was 50 times faster than the original setup. Python doesn't do bash's job very well, and bash doesn't do Python's job very well either.

Re:When Bash scripts bite by growse on Friday May 12, 2017 @05:46AM (#14322588)
I agree that for the simple job of "start process, maybe echo some stuff", then bash is superior to python. However, my difficulty always arises the moment you want to start writing 'if', or handle some sort of error condition.

Once you go down that road, you're into magic switches at the top of your script (which TFA demonstrates aren't well understood), and subtle control issues where bash's flow control doesn't quite do what you think it should, and (for me) this is where bash stops being superior.

Re:When Bash scripts bite by arghwhat on Friday May 12, 2017 @06:19AM (#14322731)
I don't think flow control is a problem, as long as it's just to complete a simple purpose. Checking if a file is present, running some code for every file or every line, etc.

But bash is definitely not something where complicated logic belongs.

Re:When Bash scripts bite by edejong on Friday May 12, 2017 @05:51AM (#14322614)
Although it's been a long time since I last did perl programming, it has always been my understanding that perl is particularly well designed for piping and mangling input and output from binaries. It's been the successor of awk, sed, grep and cut, all contained in one, relatively consistent and well documented language. Are you saying Larry Wall failed in this design attempt? Could it be that Perl wasn't used to its fullest potential?
Re:When Bash scripts bite by mfontani on Friday May 12, 2017 @06:14AM (#14322705)
Perl is great at "mangling input", absolutely.

But doing stuff like executing commands with standard output or standard error redirection, with pipes, etc. is something it's not strong at.

That's what a shell script is strong at.

To clarify, I'd: use bash to create an execution pipeline, which itself may contain a Perl script which munges some output.

Bash in this case is more of a "pipeline and execution orchestrator", and Perl is used to "properly munge the output" to make stuff happen.

Re:When Bash scripts bite by jerf on Friday May 12, 2017 @09:36AM (#14323820)

    use IPC::Run qw(run);
    run(["cat", "/etc/passwd"], "|",
        ["egrep", "-v", ':/bin/bash$'], "|",
        ["wc", "-l"],
        undef, \$out);
    print $out;
Not the best way to do that sort of thing (useless use of cat for instance), I was just trying to come up with an executable example. run returns success or failure of the pipeline as the return code, which I don't show here. I don't think you can pick out which command failed.

I also find that once you're in a real programming language, you have much less need to have deep pipelines anyhow. Consider the pure-perl equivalent

    open PW, "<", "/etc/passwd" or die;
    while (<PW>) {
        next if $_ =~ m|:/bin/bash$|;
        $count++;
    }
    print "$count\n";
Of course "real perl" would suggest some form of "use strict" and the corresponding need to declare the $count var, but if we're discussing shell script contexts it is at least fair to consider ignoring that, since it would be a silly objection that the perl equivalent doesn't turn on checks that shell scripting doesn't even have.

But if you're in a real programming language you don't need to shell out to wc just to get a line count, or write complicated awk scripts, etc. It's much more common for me in those cases to execute a single program and control its STDIN/STDOUT/STDERR and watch its exit code.

Re:When Bash scripts bite by arghwhat on Friday May 12, 2017 @10:21AM (#14324142)
I am very much against ever shelling out from an application. Either write a shell script, or a proper application. Nothing bugs me more than shell scripts written in Python, Perl, or even gasp C++ (he was later fired).

However, most bash scripts I interact with will take at least 2-4 times the amount of code to replicate as a proper application. The line count example is a good pro-shell one, with 6 not particularly nice lines of code replacing a single, simple line:

    egrep -v ':/bin/bash$' /etc/passwd | wc -l
"Complicated" AWK scripts are also much more compact than their equivalent. Say, `xyz | awk '/thing/ { getline; print $0; }'` to get the line after a pattern. I wouldn't write an AWK script much more complicated than that, though, and for me, Perl and AWK belong in the same bucket.
Re:When Bash scripts bite by jerf on Friday May 12, 2017 @11:41AM (#14324958)
My objection to shell is that it obtains that line advantage by cheating. It has terribly poor error handling, it uses an incredibly sloppy serialization format that frequently breaks at the scales people attempt to apply it at, its performance characteristics are terrible, its debugging story is nonexistant, perhaps worst of all it affords a programming style that blinds people to how much compromise they are making with their data because of the aforementioned serialization format, and on it goes. Of course programming languages can't get as concise as shell; even the sloppiest programming language communities there are have the bar for quality set substantially above what shell can meet.

Your example isn't even equivalent to my rather sloppy Perl, since, for instance, it will do something quite different if /etc/passwd doesn't exist. Shell only has the size advantage as long as everything goes perfectly correctly.

Shell is only good for two cases: 1. You don't much care about the integrity of your input or output, and you don't much care about what happens if something goes wrong. 2. When you don't care about the aforementioned issues because you're right there on the spot and can fix things if they do go wrong, because you're using shell interactively.

Having fiddled around with designing a "new shell" every so often my current conclusion is that there is no way to bridge the gap between the optimal interactive use shell and the optimal shell script with one language, and modern shell scripting, for all the features they have apparently for shell scripts, are firmly for the interactive case.

Re:When Bash scripts bite by arghwhat on Friday May 12, 2017 @02:20PM (#14326430)
My example is still very well defined, and very easy to reason about. It will output '0' on stdout if no file is found, and an error message on stderr.

Bash is very good for the things its good at, but you still need to know what you're doing to not fuck it up. Some people don't know how to consider error handling when they code, but that's the fault of the programmer.

Re:When Bash scripts bite by jerf on Friday May 12, 2017 @03:41PM (#14327211)
I used to believe that. I don't any longer. Your second paragraph basically agrees with me that bash affords bad code that ignores errors. I think it's past time for the programming world to take that seriously. We've had 60-some years of programmers insisting they can heroically just remember all the things they need to do to write correct code. And they are observably wrong. To disagree with me on that is basically to assert that software is nearly universally of a high quality and generally robust to all reasonably forseeable error conditions.

Shell goes even beyond that, though; as evidenced by articles like this you can't even get good agreement on how you should write safe shell. When even experts can't agree on what's safe, that's just something unsuitable to any serious task.

Re:When Bash scripts bite by arghwhat on Friday May 12, 2017 @05:50PM (#14328225)
None of my previous nor future paragraphs agree with you. You must have misread.

You are presenting a false analogy. I was not, am not and will never argue that one needs to be an oracle that foresees all errors (which would be absurd), but I am arguing that poor error handling is a programmer error due to the situation not being Bash specific. To reword my statement from the previous comment: If your error handling in Bash is insufficient, your error handling would also be insufficient in most other languages with implicit error handling (e.g., those with unchecked exceptions).

Once again, your Perl blob is a great example. You manually added the optional `or die` to deal with the case where the file wasn't present - perl's equivalent of `|| exit 1`. If you had not remembered to add that check, then your program would fail silently, with $count being an undefined variable. You could try to enforce better checking in the local block scope with some hacky header at the beginning of the script ("use strict;"), but that wouldn't stop you from having poor code in a module, and it would only ensure that count was defined, not that the program was well-behaved. That sounds awfully sloppy, doesn't it? Almost like what you were complaining about for Bash!

Very few languages provide you with any assistance to remind your of error checking (failing either silently or by crashing), mostly due to the awful concept of exceptions (unchecked, specifically, but checked exceptions are only useful if unchecked don't exist). Go, Rust, and - ironically - C helps you by forcing you to think about error handling at every single call-site, with lazy error handling sticking out like a sore thumb (explicitly ignored return values, which look weird in these languages), while C++, Java, Perl, Python and Bash all expect you to decide what you feel like handling today.

Re:When Bash scripts bite by omribahumi on Friday May 12, 2017 @04:58AM (#14322403)
How is bash faster than Python?
Re:When Bash scripts bite by arghwhat on Friday May 12, 2017 @06:11AM (#14322686)
When what you need is a pipe, Python won't do anything positive to performance. Plus, CPython is horrendously slow in general.

Other than that, the simpler design also allowed greater parallelism.

Re:When Bash scripts bite by barrkel on Friday May 12, 2017 @09:22AM (#14323721)
Bash lets you glue best-of-breed tools together, as long as they all talk on stdin / stdout / files. The tools themselves can be written in the best language for their task.