This is an important distinction, and probably why I still make shell scripts.
I've found utilities like `shellcheck` invaluable for making robust scripts and maintaining them.
Possible? Maybe. Easy? No. Especially the “testable” part.
The vast majority of engineers do not know how to write clean shell, let alone testable shell. There’s a reason for this: it’s extremely difficult and non-obvious to do, and generally isn’t worth investing the time in learning (you can just use a different language instead: perhaps one with support for advanced features like… arrays…)
So sure, you can write your 500 line shell script and it may well end up being beautiful. But what happens in 5 years when someone who isn’t as enlightened as you has to maintain it?
(This all without mentioning the worst aspect of shell scripting: security. Good luck handling secrets without leaking them in one way or another).
Second, yes, I do think the 5000 line Python script would be more maintainable than than the 500 line sh script. I'd choose real types and data structures any day of the week.
a testable shell script? Never seen one.
Thinking about scripts I've read in the past, I remember seeing Jason Donenfeld's bash script for wireguard-wg and thinking how productive and readable it was,
https://github.com/WireGuard/wireguard-tools/blob/master/src...
i dunno about testing it, i feel like that’s getting pretty close to the barrier of shell’s utility
The dehydrated ACME client is 2400 lines of bash/zsh:
* https://github.com/dehydrated-io/dehydrated
And its external dependencies are OpenSSL and cURL. The acme.sh shell ACME client is 8000 lines of shell:
* https://github.com/acmesh-official/acme.sh
The official Let's Encrypt client is written in Python, and the core 'executable' is much longer, and in addition it pulls in a boatload of dependencies:
https://git.9front.org/plan9front/plan9front/HEAD/sys/src/cm...
* https://github.com/diafygi/acme-tiny/blob/master/acme_tiny.p...
Long time ago some dude created a couple of korn shell scripts on Aix to automate file uploads and ETL related tasks.
With time that work moved from Aix into Red-Hat Linux, still korn shell, with the adaptations from Aix into GNU/Linux command line parameters and such.
I got tasked to port them into Java, because the team that got the job to maintain the servers lacked UNIX experience, and they thought selling the porting project was a good idea.
Feedback from customer, slower than those scripts and not as easy to change, what a surprise.
consider this sentence:
"arrays do not exist in posix shell"
...
"NOOO!" you shout, because you like data structures a LOT
"now THAT'S a good reason to use bash! bash supports arrays!"
absolutely not!
i believe that you should use shell if your problem is:
- small and scoped (~200 lines of shell or less)
- unlikely to increase in size and scope as time goes on
- not very complex
if you find yourself desiring arrays, good error handling,
static typing, structs, etc, then your problem should
be solved using a different language, not shell.
It's also why you see people bothering to use perl 5 at all in this day and age. It just happens to be preinstalled on a decent majority of *nix systems.
A young traveller came to visit Emperor Sh, and found him sitting in his
sparsely furnished temple.
“Emperor Sh,” he said, “I am told you are the greatest scholar of shell that
the world has known.”
Emperor Sh made no reply. The traveller continued.
“I have come to ask your advice. I am thinking of developing a character-based
graphing tool. It will interactively change the plot based on key presses.
Which shell commands should I use?”
“Don’t do it in shell,” said Emperor Sh, curtly.
The young traveller was confused. He tried again. “Well… I am also working on
a database audit script. It needs to verify that certain characters do not
appear in any fields in several tables. What should—”
“Don’t do it in shell,” interrupted Emperor Sh.
The traveller began to despair. “I have journeyed one thousand miles, tried
thirteen distributions of my operating system, and waded through hundreds of
badly-written manual pages,” he cried, “and now I have finally come to Emperor
Sh, the greatest shell programmer in the world, and I am told to use no shell
at all! Perhaps I should do what my Python user friends told me to do, and
just pipe together some scripts in venv environments!”
“Good idea,” said Emperor Sh. “Do it in shell.”
Enlightenment crushed down on the young traveller and he bowed to the Emperor,
sobbing with reverent joy.
I’m usually dealing with other people’s python, so I don’t remember which ones broke. Usually stuff around 3rd party services, async I/O or parsing.
I would add a 4th condition: you aren't going to share the script with anyone else (including CI) - it's going to stay strictly on your computer and nobody else will ever read it.
I never got into Perl, but I understand why it was popular.
Sure it’s possible, but it’s therefore also brittle, when someone touches it and is not a complete expert.
Shell is nice not because of the shell language, but the ease of executing and chaining other programs.
Many tasks are more concisely expressed and readable in shell than in any other language.
I'm doing programming language research and it's just infuriating how retrograde shell is. Its design is a fossil of history, and boy did our understanding of what makes a good language evolve. As a proof of this look at the jq tool. It is so powerful and clean. The implementation is simple and reasonably fast. It has been designed by a researcher from functional languages (an ocaml guy). This is a way to build a language around pipelining. At a lesser level, look at how moving from vimscript to lua in neovim has brought so many high-quality new plugins, look at how many people now tinker with low-level stuff and write high-perf libraries now that C/C++ is not the only choice. Surface language and having very little very orthogonal concepts is important.
Some core implementation patterns for PL are very old, but there have been lots of simplifications and rationalizations in surface language. Error handling, values hygiene, scoping, simple datastructures. Posix shell might have been a prowess in the 70s, now it is mostly a hack and this is a fact. Let's be a bit less superficial when honoring the unix mantras and dream about something better (making a better shell for its core ideas(!), that is, handling processes and fds).
As it passes objects the filtering, sorting, joining is generic across files or any other data type.
I really cannot fathom how this could be, as Microsoft bundles curl, which has nearly 100 exit codes on various types of failure.
Whoever made this language decision needs to explain themselves.
Use $LASTEXITCODE for native executables, and $? for PowerShell cmdlets.
https://learn.microsoft.com/en-us/powershell/module/microsof...
It’s untyped, untestable, arcane, and unmaintabale. Even the link says don’t do this. Just don’t.
Congrats on having the chops (and neckbeard) to pull this off. If you want to do it in a personal project, more power to you. But it doesn’t scale at organizations.
So, it depends on the context. "Organization" is a very broad term. If the team you are in is familiar with the shell, and the shell is the right tool for the job, there isn't that much of a problem.
I'm not arguing for writing large shell scripts when there are better alternatives. But, sometimes, it's the right choice and it can be written properly. And even if it isn't... 500 lines of poorly written shell aren't that many in any case, and they aren't that bad compared to 5000 lines of poorly written Python for example.
And the problem with bash is similar to C: many people think they can write correct C code, but as it has been shown plenty of times, that doesn’t really work out in the end. Languages have different tradeoffs and bash really has no redeeming qualities, imo.
Because Bourne shell is not a Python-class language like Ruby or perhaps even Scheme or AWK are. Things that are easy in shell can be hard in Python and vice versa: awkward wrangling of subprocess.Popen in Python can be a single line of shell; simple tree munging with lxml in Python can easily be a hundred or more lines of wasteful xmlstarlet invocations in shell. Parallel processing, even without the heavy artillery of make or GNU parallel, is easy in shell but hard in Python. Structured data is hard in shell but easy in Python.
(Similarly, I wouldn’t use ML or Haskell to write a web scraper, and I wouldn’t use Python to write a compiler.)
Perhaps a better way to put it would be that sloppy concurrency is easy. This doesn’t sound all that good, but it’s easy enough that I find myself parallelizing one-off things in shell that would be such a hassle to do nonsequentially in Python (compared to the magnitude of the problem) that the option wouldn’t even surface in my brain. (If I rewrite the shell script in Python later, the rewritten version is often sequential.)
The shell error handling story for “run stream processing steps in lockstep” parallelism is much better than for “spawn a bunch of identical children” parallelism, though, and that’s often enough.
#!/bin/bash -eux -o pipefail
TMPDIR=$(mktemp -d tempdirXXXXX)
trap “rm -rf $TMPDIR” SIGINT SIGTERM ERR EXIT
(
cd $TMPDIR
foo | grep x > foo.out &
bar > bar.out &
baz $TMP | tee baz.out &
)
wait
tar -cf - -C $TEMPDIR | zstd -6 -z - | pv | ssh server.com bash -c “cat - > out.tag.zstd” ... | ssh server.com dd of=out.tag.zstd $ shellcheck myscript
Line 1:
#!/bin/bash -eux -o pipefail
^-- SC2096 (error): On most OS, shebangs can only specify a single parameter.
Line 4:
trap "rm -rf $TMPDIR" SIGINT SIGTERM ERR EXIT
^-- SC2064 (warning): Use single quotes, otherwise this expands now rather than when signalled.
Line 6:
cd $TMPDIR
^-- SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean: (apply this, apply all SC2086)
cd "$TMPDIR"
Line 9:
baz $TMP | tee baz.out &
^-- SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean: (apply this, apply all SC2086)
baz "$TMP" | tee baz.out &
Line 13:
tar -cf - -C $TEMPDIR | zstd -6 -z - | pv | ssh server.com bash -c "cat - > out.tag.zstd"
^-- SC2086 (info): Double quote to prevent globbing and word splitting.
^-- SC2153 (info): Possible misspelling: TEMPDIR may not be assigned. Did you mean TMPDIR?
Did you mean: (apply this, apply all SC2086)
tar -cf - -C "$TEMPDIR" | zstd -6 -z - | pv | ssh server.com bash -c "cat - > out.tag.zstd"Came here to say this.
At one end you have things like Haskell, OCaml, Rust, (or if you go really extreme Dafny etc.).
At the other end you have Bash.
There's no way a 500 line Bash script would require 5000 lines of Python.
Also, to understand someone else’s python, you have to read at least 10x more lines of line noise, so it somehow manages to be less maintainable than shell.
Concrete example: Every python code base I’ve encountered has some function named exec that poorly reimplements shell job control and argument passing. As the python code bases grow, they end up with more than one of these, all of which have different bugs.
I usually discover this about 6 hours after the script has been sent to me, since someone who has heard the words “supply chain vulnerability” locked down whatever machine I need to run the script on.
The last ten times this has happened (at multiple companies, with multiple teams) it ended up being faster to figure out what shell commands the python script wanted to invoke, port a few things to jq and then reimplement the rest in a mix of shell and perl.
The result is always 1-10% the lines of code of the python, and actually runs without downloading an entire python distribution, dependency manager, and hundreds of broken packages at runtime.
Completely untrue. You're conflating static typing with strong typing. And pytest is a great test framework.
And you're recommending shell for better typing and testability? The language where everything is a string and there are no proper arrays? You have to be trolling.
pytest is terrible, at least from how I’ve seen people use it. It hides the diagnostic error explaining why the test failed in a random place. Its output is not parseable. If you have a long running test that times out, then it buffers the stdout and stderr so you have to wait for the fallback test time out every. single. time.
Mypy has had strict Optional checking on as default since 2018, almost entirely eliminating any None-related errors, meaning it is actually stricter and safer than Java or other traditionally considered static languages in that regard, (bar any dynamic misuse in deserialization boundaries). Assuming you use mypy of course, but that's a given today imo, like using shellcheck is for sh.
Then one would simply choose a proper language that had more of those solid principles laid at its foundation
"Stephen Bourne's coding style was influenced by his experience with the ALGOL 68C compiler that he had been working on at Cambridge University... Moreover, – although the v7 shell is written in C – Bourne took advantage of some macros to give the C source code an ALGOL 68 flavor. These macros (along with the finger command distributed in Unix version 4.2BSD) inspired the International Obfuscated C Code Contest (IOCCC)."
Agreed on avoiding rewrites if possible but 1/10 ratio is laughable.
In my opinion, shell scripts should be used either as glue or due to portability reasons. Rewriting a perfectly good and working script just because makes no sense.
So if the script started as some quick and easy way to deploy to prod but slowly becomes a massive cli "one tool to rule them all" kind of thing.. maybe it's time to change course.
I'm proficient at POSIX shell scripting and use it all the time, but every time I discover that a tool I need to maintain or, god forbid, improve upon is written in pure shell I immediately get a migraine.
The shell is not a real language and must be written as such.
Take a look at winetricks, or clitest, or crossroot-ng, or shellspec. They are modular, easy to read, well written pieces of software that are written in shell.
In the 2000s, people used to say that it was impossible to write large stuff in JavaScript, and it was not a real language. They were wrong, because they only saw browser differences and the issues with it, not the great things about it. The bourne shell is in a similar state.
I'm not saying that it's impossible to do, I'm saying that it's very difficult, requires every contributor to be extremely well versed into shell coding and the particular convention of the codebase, and small issues can easily snowball out of control. It's doable, but it's almost never worth it IMO. One notable exception is when you need max portability with effectively zero dependencies.
>In the 2000s, people used to say that it was impossible to write large stuff in JavaScript, and it was not a real language.
They were partly right, which is why modern web development bears very little resemblance with early 2000's JS hacking. Remember that even JQuery only appeared in 2006. And those were the dark days of IE6...
But IMO that's not exactly the same thing: the issue with Javascript was less with its syntax and more with the very poor Javascript API and standard lib (or lack thereof), shell scripting is the other way around in a sense: you have access to all the power of the OS and all the programs in it, but it's the actual programming language that's lacking.
Actually a decent language that would transpile to POSIX sh would be interesting, now that I think about it.
For example, all moderns shells support some form of `local` for declaring local variables. There are some small subtleties but consistent behavior can be achieved across interpreters. It's not on POSIX though. Shellspec complaints about it (but it shouldn't).
It sounds small, but it's a huge thing to have local variables instead of a huge global scope.
There are many other features that are widely available but with slightly different interfaces. Kind of what happened with ActiveX/XmlHttpRequest in JavaScript.
What is missing is precisely documentation on the subset of stuff that _can_ be portable with coding standards, polyfills and adapters.
I'm not talking about core utilities here. No `cat` or `grep` or nothing like it. Basially, `PATH=''`, only builtins are allowed. Even with this constraint, the shell is usable and powerful.
Unfortunatelly, most people are well versed in bash+coreutils, not raw portable shell. It requires learning a few things that are not straightforward at first (variable substitution, how the shell forks processes, making evals without making a mess, nested quoting, etc).
> Actually a decent language that would transpile to POSIX sh would be interesting, now that I think about it.
That would be nice. I believe it can be written in shell itself, self-hosted and portable. It's hard work though (on the same level of writing babel from scratch).
Isn't this how we got the ultraportable monster configure scripts? Not sure about "decent" though...
Yes. Shell scripts can be terrible, and most are. But if one approaches them thinking that “the shell is a real language and must be written as such”, with solid principles in mind, it is perfectly possible to write clean, maintainable, and testable shell scripts.
Oh, and that 500-line shell script probably ends up being a 5000-line Python monster anyway.