News for hackers. Stuff that matters.
Re:When Bash scripts bite by therealmarv on Friday May 12, 2017 @03:39AM (#14322103)
When I see very long bash scripts I get angry. There is a reason why perl & later python were invented and they are both in 98% of cases available on Unix compatible systems.

Not that I'm not ok with short bash scripts or for systems were you only have bash... but if your Unix compatible system only has bash there is something else wrong with your system in my point of view.

Re:When Bash scripts bite by arghwhat on Friday May 12, 2017 @04:43AM (#14322341)
I'd much rather have 200 lines of bash than the 300-400 lines of python to replace it.

I have recently rewritten a lot of Python and Perl (over 10kLOC) to a small collectiom of 10-ish bash scripts that weigh in at about 700LOC total. There are some tasks where Bash is the only appropriate tool.

Much simpler code, much easier to read, and a lot faster too.

Re:When Bash scripts bite by growse on Friday May 12, 2017 @04:59AM (#14322406)
If you're measuring code quality / maintainability by how many lines of code there are, you're doing it wrong. I'd much rather see 400 lines of well-structured python than 200 lines of tangled bash. For the rewrite you did, how much of the benefit was because it was rewritten, rather than being rewritten _in bash_?

I'd also go as far as saying that if you're using bash because it's faster than python, you're also using the wrong tool for the job.

Re:When Bash scripts bite by arghwhat on Friday May 12, 2017 @05:38AM (#14322558)
The quality of the bash far outshined the quality of the python and perl, because Python and Perl was being used to do a bash scripts job. When what you need is to call a bunch of binaries and redirect output around, then Python and Perl just don't cut it.

The code is much, much easier to read now. You can see a script on a single screen height, which works wonders for overview. The 3x performance gain spawned from enlightenment this overview granted us.

But don't get me wrong, in the very same rewrite, I rewrote 100 lines of bash + 300 lines of Python into a single 200 lines of Python, which when combined with PyPy, was 50 times faster than the original setup. Python doesn't do bash's job very well, and bash doesn't do Python's job very well either.

Re:When Bash scripts bite by growse on Friday May 12, 2017 @05:46AM (#14322588)
I agree that for the simple job of "start process, maybe echo some stuff", then bash is superior to python. However, my difficulty always arises the moment you want to start writing 'if', or handle some sort of error condition.

Once you go down that road, you're into magic switches at the top of your script (which TFA demonstrates aren't well understood), and subtle control issues where bash's flow control doesn't quite do what you think it should, and (for me) this is where bash stops being superior.

Re:When Bash scripts bite by arghwhat on Friday May 12, 2017 @06:19AM (#14322731)
I don't think flow control is a problem, as long as it's just to complete a simple purpose. Checking if a file is present, running some code for every file or every line, etc.

But bash is definitely not something where complicated logic belongs.

Re:When Bash scripts bite by edejong on Friday May 12, 2017 @05:51AM (#14322614)
Although it's been a long time since I last did perl programming, it has always been my understanding that perl is particularly well designed for piping and mangling input and output from binaries. It's been the successor of awk, sed, grep and cut, all contained in one, relatively consistent and well documented language. Are you saying Larry Wall failed in this design attempt? Could it be that Perl wasn't used to its fullest potential?
Re:When Bash scripts bite by mfontani on Friday May 12, 2017 @06:14AM (#14322705)
Perl is great at "mangling input", absolutely.

But doing stuff like executing commands with standard output or standard error redirection, with pipes, etc. is something it's not strong at.

That's what a shell script is strong at.

To clarify, I'd: use bash to create an execution pipeline, which itself may contain a Perl script which munges some output.

Bash in this case is more of a "pipeline and execution orchestrator", and Perl is used to "properly munge the output" to make stuff happen.

Re:When Bash scripts bite by jerf on Friday May 12, 2017 @09:36AM (#14323820)

    use IPC::Run qw(run);
    run(["cat", "/etc/passwd"], "|",
        ["egrep", "-v", ':/bin/bash$'], "|",
        ["wc", "-l"],
        undef, \$out);
    print $out;
Not the best way to do that sort of thing (useless use of cat for instance), I was just trying to come up with an executable example. run returns success or failure of the pipeline as the return code, which I don't show here. I don't think you can pick out which command failed.

I also find that once you're in a real programming language, you have much less need to have deep pipelines anyhow. Consider the pure-perl equivalent

    open PW, "<", "/etc/passwd" or die;
    while (<PW>) {
        next if $_ =~ m|:/bin/bash$|;
        $count++;
    }
    print "$count\n";
Of course "real perl" would suggest some form of "use strict" and the corresponding need to declare the $count var, but if we're discussing shell script contexts it is at least fair to consider ignoring that, since it would be a silly objection that the perl equivalent doesn't turn on checks that shell scripting doesn't even have.

But if you're in a real programming language you don't need to shell out to wc just to get a line count, or write complicated awk scripts, etc. It's much more common for me in those cases to execute a single program and control its STDIN/STDOUT/STDERR and watch its exit code.

Re:When Bash scripts bite by arghwhat on Friday May 12, 2017 @10:21AM (#14324142)
I am very much against ever shelling out from an application. Either write a shell script, or a proper application. Nothing bugs me more than shell scripts written in Python, Perl, or even gasp C++ (he was later fired).

However, most bash scripts I interact with will take at least 2-4 times the amount of code to replicate as a proper application. The line count example is a good pro-shell one, with 6 not particularly nice lines of code replacing a single, simple line:

    egrep -v ':/bin/bash$' /etc/passwd | wc -l
"Complicated" AWK scripts are also much more compact than their equivalent. Say, `xyz | awk '/thing/ { getline; print $0; }'` to get the line after a pattern. I wouldn't write an AWK script much more complicated than that, though, and for me, Perl and AWK belong in the same bucket.
Re:When Bash scripts bite by jerf on Friday May 12, 2017 @11:41AM (#14324958)
My objection to shell is that it obtains that line advantage by cheating. It has terribly poor error handling, it uses an incredibly sloppy serialization format that frequently breaks at the scales people attempt to apply it at, its performance characteristics are terrible, its debugging story is nonexistant, perhaps worst of all it affords a programming style that blinds people to how much compromise they are making with their data because of the aforementioned serialization format, and on it goes. Of course programming languages can't get as concise as shell; even the sloppiest programming language communities there are have the bar for quality set substantially above what shell can meet.

Your example isn't even equivalent to my rather sloppy Perl, since, for instance, it will do something quite different if /etc/passwd doesn't exist. Shell only has the size advantage as long as everything goes perfectly correctly.

Shell is only good for two cases: 1. You don't much care about the integrity of your input or output, and you don't much care about what happens if something goes wrong. 2. When you don't care about the aforementioned issues because you're right there on the spot and can fix things if they do go wrong, because you're using shell interactively.

Having fiddled around with designing a "new shell" every so often my current conclusion is that there is no way to bridge the gap between the optimal interactive use shell and the optimal shell script with one language, and modern shell scripting, for all the features they have apparently for shell scripts, are firmly for the interactive case.

Re:When Bash scripts bite by arghwhat on Friday May 12, 2017 @02:20PM (#14326430)
My example is still very well defined, and very easy to reason about. It will output '0' on stdout if no file is found, and an error message on stderr.

Bash is very good for the things its good at, but you still need to know what you're doing to not fuck it up. Some people don't know how to consider error handling when they code, but that's the fault of the programmer.

Re:When Bash scripts bite by jerf on Friday May 12, 2017 @03:41PM (#14327211)
I used to believe that. I don't any longer. Your second paragraph basically agrees with me that bash affords bad code that ignores errors. I think it's past time for the programming world to take that seriously. We've had 60-some years of programmers insisting they can heroically just remember all the things they need to do to write correct code. And they are observably wrong. To disagree with me on that is basically to assert that software is nearly universally of a high quality and generally robust to all reasonably forseeable error conditions.

Shell goes even beyond that, though; as evidenced by articles like this you can't even get good agreement on how you should write safe shell. When even experts can't agree on what's safe, that's just something unsuitable to any serious task.

Re:When Bash scripts bite by arghwhat on Friday May 12, 2017 @05:50PM (#14328225)
None of my previous nor future paragraphs agree with you. You must have misread.

You are presenting a false analogy. I was not, am not and will never argue that one needs to be an oracle that foresees all errors (which would be absurd), but I am arguing that poor error handling is a programmer error due to the situation not being Bash specific. To reword my statement from the previous comment: If your error handling in Bash is insufficient, your error handling would also be insufficient in most other languages with implicit error handling (e.g., those with unchecked exceptions).

Once again, your Perl blob is a great example. You manually added the optional `or die` to deal with the case where the file wasn't present - perl's equivalent of `|| exit 1`. If you had not remembered to add that check, then your program would fail silently, with $count being an undefined variable. You could try to enforce better checking in the local block scope with some hacky header at the beginning of the script ("use strict;"), but that wouldn't stop you from having poor code in a module, and it would only ensure that count was defined, not that the program was well-behaved. That sounds awfully sloppy, doesn't it? Almost like what you were complaining about for Bash!

Very few languages provide you with any assistance to remind your of error checking (failing either silently or by crashing), mostly due to the awful concept of exceptions (unchecked, specifically, but checked exceptions are only useful if unchecked don't exist). Go, Rust, and - ironically - C helps you by forcing you to think about error handling at every single call-site, with lazy error handling sticking out like a sore thumb (explicitly ignored return values, which look weird in these languages), while C++, Java, Perl, Python and Bash all expect you to decide what you feel like handling today.

Re:When Bash scripts bite by omribahumi on Friday May 12, 2017 @04:58AM (#14322403)
How is bash faster than Python?
Re:When Bash scripts bite by arghwhat on Friday May 12, 2017 @06:11AM (#14322686)
When what you need is a pipe, Python won't do anything positive to performance. Plus, CPython is horrendously slow in general.

Other than that, the simpler design also allowed greater parallelism.

Re:When Bash scripts bite by barrkel on Friday May 12, 2017 @09:22AM (#14323721)
Bash lets you glue best-of-breed tools together, as long as they all talk on stdin / stdout / files. The tools themselves can be written in the best language for their task.
Re:When Bash scripts bite by gjulianm on Friday May 12, 2017 @04:08AM (#14322214)
Python and perl are not always the best option. For example, if I am executing a lot of system commands (sometimes through ssh), checking outputs with grep/awk, reading files, calling make, why use a language that is going to be far more verbose, difficult to modify, and show problems because it's not executing the shell in the correct profile or things like that? For interacting with the system, there's nothing better than shell scripts. You mainly only have to take care with the quotes to avoid word separation and globbing, and use the *nix commands to avoid recreating the wheel.
Re:When Bash scripts bite by scaryclam on Friday May 12, 2017 @05:37AM (#14322552)
I'll second this. My Python is far better than my bash but there's no point writing more complicated Python if you're working with system commands. That said, quite often I'll break non-system stuff out into python scripts and wrap them in bash if I need to do a bit of both.
Re:When Bash scripts bite by arjie on Friday May 12, 2017 @05:27AM (#14322507)
People always recommend Perl over a shell script but I've never seen a situation where it makes sense. Having written very little Perl, it seems like its only strength is that it's present everywhere and has a large library.

Language-wise, Python is far superior and has a large ecosystem and these days it has ubiquity too.

I'd be interested in the argument for Perl over shell scripts that does not apply to just writing the program in a modern language.

Re:When Bash scripts bite by kagamine on Friday May 12, 2017 @05:48AM (#14322597)
the problem with Python where I work is that developers are openly hostile toward it. We're predominantly .NET with just a couple of products running on Linux servers. I'm not a .net guy despite the pressure, and have been teaching myself python in my own time, but the the rest of the staff including management are never going to look for Python as a skill when hiring. EVER. But of course 2.7 is sitting there on the CentOS servers never used, while bash scripted cron jobs run amok and frequently fail (because they were written by the wrong people imo).

I would love to see Python utilized, and maybe with VSCode and MS integrating open source into the .NET world it will eventually come to enterprise (where I work).

Re:When Bash scripts bite by therealmarv on Friday May 12, 2017 @05:55AM (#14322628)
It's already there in the big Visual Studio: https://www.visualstudio.com/vs/python/

And it's a dream in VSCode + the most popular python extension https://github.com/DonJayamanne/pythonVSCode

Re:When Bash scripts bite by kagamine on Friday May 12, 2017 @06:32AM (#14322780)
it's not about what MS put in VS and VSCode, it's about changing the minds of .NET developers who used are to C# syntax/grammar and hate Python because it doesn't fit their world-view. "Already in VS" is meaningless in a .NET enterprise company.
Re:When Bash scripts bite by therealmarv on Friday May 12, 2017 @06:46AM (#14322834)
I feel sorry for you. Fanatism towards any language/eco-system, no matter if .net, Java, Ruby, Python (also have seen the same with Python developers) is something I really don't like.
Re:When Bash scripts bite by therealmarv on Friday May 12, 2017 @05:36AM (#14322543)
Have you ever done string manipulation (also RegEx) over many files in Perl? I think it's in this regard superior than anything else, even python when looking at syntax.
Re:When Bash scripts bite by arjie on Friday May 12, 2017 @07:00AM (#14322881)
I have not. I usually use `sed` in such instances but admitting the superiority of Perl in such instances surely this is insufficient on its own to use Perl? Seems like a rare use-case and if that script grows large then Python has superior language features but if the script stays small, then a Bash script with `sed` will do just as well.
Re:When Bash scripts bite by Outpox on Friday May 12, 2017 @05:54AM (#14322625)
For someone who hasn't done any Perl yet and only little Python could you explain why?
Re:When Bash scripts bite by mfontani on Friday May 12, 2017 @06:19AM (#14322729)
For the specific thing the OP mentioned, this:

    $ perl -pi -e 's/MY_NAME/Marco/g' ./*.txt
... would go through all "*.txt" files in the directory, and globally replace the token "MY_NAME" with "Marco".

Great for a {poor man's,simple and easy to understand} basic template system to be used inside a bash script ;)

Re:When Bash scripts bite by JetSpiegel on Friday May 12, 2017 @07:47AM (#14323115)
Overkill. Just use:

    $ sed -i 's,MY_NAME,Marco,g' *.txt
Re:When Bash scripts bite by therealmarv on Friday May 12, 2017 @09:05AM (#14323617)
nope. I've dealt recently with some big DB files (I think GB size) and regex doing the same. sed is a lot slower than perl for this kind of task.
Re:When Bash scripts bite by jstimpfle on Friday May 12, 2017 @09:55AM (#14323953)
I don't know how sed and perl compare, but I know these things:

Use a C local (e.g. LANG=C or LC_ALL=C) for dramatic speed up of tools like grep/sort and probably also sed. If you don't do that, usually the UTF-8 parser kicks in, which is a lot slower.

Perl is actually well known to have a non-optimal regex implementation: https://swtch.com/~rsc/regexp/regexp1.html That said, I don't know how fast it is (what optimizations it has) at simple string substitutions.

Re:When Bash scripts bite by yjftsjthsd-h on Friday May 12, 2017 @07:22AM (#14322987)
This is the first time I have seen anoybody hold up perl as the superior option. I don't doubt that it will work, but I need the next admin to be able to read my code.

Also, every system may have some Python installed, bud by the time I've dealt with versions (2.6 is the CentOS 6 default, 3.6 is the current version) and dependencies (virtualenv? system wide pip? PYTHONPATH? RPM Python modules?) I'd better get a significant improvement.