I work with some deployment systems that chain together small scripts in a bunch of different languages. They are a nightmare to troubleshoot.
I'd much rather the spaghetti was all on one plate than follow it from table to table...
The biggest mistake I see people making is to hard code paths, ports, user names, configuration, etc. inside Python/Perl scripts. All that stuff belongs in shell. All the logic and control flow goes in the scripts. If it's factored this way, then you have a very natural way of testing the scripts with test parameters (i.e. not production parameters).
I don't doubt that there are many multi language shell scripts that are spaghetti. Factoring into processes is definitely a skill that takes thought and effort, and I don't think anyone really teaches it or writes about it. The only books I can think of know of is The Art of Unix Programming by ESR and the Unix Programming Environment.
But it's definitely made me way more productive once I started thinking like this. People say talk about the "Unix philosophy" for a reason. It's a real thing :) It's not an accident that Unix is wildly popular and has unprecedented longevity.
Verbose, and almost guaranteed to be more accurate. The thing about python/ruby/perl is that when you do things the programmatic way you get safety by default. You don't have to remember to 'find -0 | xargs -0' and thing aren't going to bite you in the butt when filenames have spaces in them.
Yes, you can make all that work with shell (heaven knows I've done it), but it's more work and sadly isn't shell's default state.
python3 - <<'END'
print('hello world')
END
The advantage is that you still have only one shell script to ship, which makes sense when the python code inside it is specific to that one script.Remember to use the <<'END' variant so variable expansion isn't done inside the heredoc. On the other hand, $ doesn't do anything in python and if you're feeling like it (please make sure everybody else is also feeling like it) you can use variable substitution there, in effect generating a python script and executing it.
As mentioned above, things that belong in shell, and NOT in Python:
- file system paths (these are parameters to Python scripts, not hard coded)
- port numbers
- user names
- other config values
- concurrency -- you can get really far with xargs -P, and sometimes &
- pipelines (try doing this in Python without shelling out; it's horrible)
Things that belong in Python and NOT shell: - complex control flow (e.g. I rarely write loops in bash; never a nested loop)
- data structures (bash has hash tables, but I never use them)
- arithmetic
- parsing and cleaning individual lines or fields of data
Basically it's policy vs mechanism. They really are mutually exclusive, and work well together.To be honest, I trust myself more manipulating paths in a language like Python than to put all quotes in the correct places in Bash :(
- Avoid paths with spaces :)
- If you have to handle paths with spaces, use double quotes everywhere,
e.g. "$pidfile".
- Always use "$@". I've never found a reason to use $@ or $* unquoted.
I also write unit tests (in shell) if there is something unusual I know I have to handle.That's it. I don't think it's hard at all, although you will certainly see scripts in the wild where the author was confused about quoting, and quoting hacks pile upon hacks. If you use a consistent style, it's very smiple.
There is a learning curve like there is with any language, but it's totally worth it. Shell is an extremely powerful language, and saves you MANY lines of Python code.
I would say Python is definitely the easiest language to learn, but bash is not harder than say JavaScript.
I'd like to add: Always use double quotes around paths. Or is there something I'm missing here?
$ ls -l "*"
ls: cannot access *: No such file or directory
Of course, you might want that exact behaviour, but sometimes you want e.g. all .table files in a path with a space. I believe $ ls -l "path with/lots of spaces/"*
is the correct solution, but it seems silly. I'm not entirely sure why variable expansion works inside double quotes but wildcard expansion doesn’t.That's because a double-quoted expression is supposed to evaluate to a single word. Wildcard expansion can result in multiple words, but parameter expansion doesn't.
(Except for things like "$@". With bash, the rules always have exceptions.)
People who insist on rewriting an ENTIRE program in Python, Perl, or Ruby fail to understand the Unix philosophy (this is a real misunderstanding I've encountered in my work, not a straw man).
You can just write the complex part in another language, but keep the rest in bash. In other words, main() stays in bash. Python et. al. is used as just another process. bash is a language for coordinating processes.
You don't want a 2000 line bash script. But it can be worse to have a 5,000 line Python script that shells out to tons of other utilities, or awkwardly and verbosely reimplements 'xargs -P' or 'sed -i'. Often you can do the same job with 500 lines of bash and 500 lines of Python (or C), and that is the ideal solution.
Python and bash are pretty radically different languages, and they are not interchangeable (the advices "just rewrite in Python" seems to imply they are). You can use each one for the things they are good at.