It's definitely possible to write shell code which properly passes shellcheck's linter though it's an uphill battle to learn the ins and outs and _why_ X is wrong when Y is right.
I even managed to write a full TUI file manager in bash!
https://github.com/dylanaraps/fff
I full understand that there are times when the shell should not be used and when other languages are a better way to solve a specific problem, however I love pushing the shell beyond its supposed limits! :)
I've read pretty much everything I could get my hands on regarding the shell (including the mentioned link) and I still love it.
> Do you write truly correct Bash/POSIX code?
If we define correct as passing shellcheck, avoiding all pitfalls and maintaining compatibility (POSIX sh not bash), then yes, I like to think so. :)
> Do you still love it?
Oh yeah! I've been writing a ton of POSIX sh as of late. My latest project being a Linux distribution: https://getkiss.org/
(hello from Firefox in KISS!)
Edit: follow up question is Do you believe bash / POSIX shells actually follow KISS principles?
Not questioning whether your OS is KISS, but I don't think that necessarily reflects the KISS-ness of the underlying language.
My questions clearly reflect my current impression that in the long term, shell pitfalls largely undermine the benefits of its apparent simplicity. The gist would be for you to provide some way to change my mind. I guess the codebase you provide is a strong counter example; but you'll agree it doesn't reflect general usage of shell in the wild.
Edit 2: You know what, I just read your original comment again. I kind of retract my question since you do concede that it's an uphill battle and you love it in spite of its flaws. I guess that's cool (and I agree it's fun trying to write correct bash as a challenge) as long as you're in control of the code being produced, but my main impression remains that it's a bad language to publicize and its presence in most codebases inherently bears a strong cost.
POSIX `sh` yes. `bash` less so but I'd still lean more towards a yes.
Ultimately though, it depends on how we define "simple". Both `bash` (2.6MB) and POSIX `sh` shells (`dash` (232KB), `ash` (1.2MB (busybox)), etc) are tiny in size if we compare them to Python (137MB) or Perl (44MB).
(Numbers taken from my system using `du` on each file which belongs to each shell/language.)
If we define "simple" to language features then I think the shells come out on top again (especially POSIX `sh`).
If we define "simple" as ease of use (without shooting yourself in the foot) then I'd agree with you and say that the shell loses here.
There's a time and place for using any tool (in production) but I find it fun to push the shell beyond what is thought possible in my personal projects. :)
It also tends to be very portable.
This is perhaps one big benefit but not one that is exclusive to a sh-like language. Instead I would like to see a language with strong flow control or metaprogramming capabilities take on processes as a first class citizen. Perl is probably the closest but still has some warts related to redirection.
The best pattern I have seen is encapsulating business logic into Python or Go and then if really necessary piping it to another script. But, often, if you do this you can just keep the piping internal using data structures.
Unix-pattern facilities work very well for interactive use, which tends to be simple, exploratory, and trial-and-error. But a project's build script may not be simple.
In fact this is how Perl 1 looks: https://st.aticpan.org/source/RCLAMP/perl-1.0_16/
https://github.com/Perl/perl5/commit/8d063cd8450e59ea1c611a2...
https://github.com/lmorg/murex
It currently has:
* Proper error handling (eg try and catch blocks)
* unit testing and debugging frameworks to help with development and maintainability
* data-type aware, including complex types like how CSV, JSON and YAML are all handled as memory structures and thus the same tools can query any structured data format without understanding it's contents
* while still ostensibly working the same way as a traditional POSIX shell
There's also some work on improving the REPL experience too where I've included:
* automatic man page parsing for flags
* a "tool tip text" like hint line which tells you where commands reside on the fs, what it does, etc. Which is handy if you're trying to debug an existing commend.
* if you paste multiline text into the console you get offered a chance to preview the text before executing it (handy if, like me, you're pretty useless at copy/pasting content reliably)
* a package management system so you know exactly which functions and imported scripts are loaded from which sources (no more "where did that autocomplete suggestion / alias / etc get loaded from?"
There's a few other features like support for events and such like, but they're not yet documented.
The shell is currently beta but I've been using it as my daily driver for about 18 months now. There are still quite a few bugs, plenty of places where code needs to be rewritten for performance and lots of stuff isn't yet documented (though the documentation is pretty good already considering it's only me working on it). So don't expect a finished product. However I do think I'm at the stage where I'm ready for more users to have a play and I welcome PRs, issues raised, general comments and feedback, etc.
# end of shameless self promotion :D
I'm aware of Elvish and really impressed with what's been built and the traction that has gained. There is definitely some overlap between murex and elvish but also a lot of area's where our shells differ.
I think there is sufficient difference between the two shells to justify their existence.
I wholehartedly agree on perl but can you expand on the redirection warts? I seldom had problems with perls' FHs, while, on the contrary I seem to be unable to wrap my head around the contorted syntax involved in bash's handling of descriptors - especially when more than 2 handles are involved.
The wart is that you need IPC::Open3 or equivalent because Perl's intrinsics can not synthesize the pipe operator (though you will think that they can) if you need to insert yourself into the middle of a chain of commands.
Nowadays there are decent wrappers for calling Open3 but more commonly you just find people running one half of the command, buffering the output, and passing it to the second half.
I quite like Powershell on Windows, but I'm not sure I dare try it on Linux. I think it might make my head explode.
As to why people like it: it often feels more natural when you are automating what you would type interactively. Unix pipes/coreutils/etc. also feel like a better fit when that automation is mostly about connecting other programs (it's the auxiliary stuff that you'd maybe want in pure bash). Reading the subprocess Python documentation does not exactly fill me with joy. I've heard libraries like Plumbum make it a bit neater - but then you have to ask why learn a bunch of new libraries when I already know bash? In the end it's about the best tool for the job. The danger with bash is going too far, especially if you don't actually know it very well.
It is unstructured only in the way you allow any running command within your script to dump their output.
I run most commands inside my scripts with `> /dev/null 2>&1` and then rely on exit codes to wrap structured information to be echoed out with this function:
echo_l(){echo;echo '--------';echo "${1}";echo '--------';echo}
Or functions such as this to populate a log:
log_cat(){echo "${1}" >> ${__LOG} }
But the tee named pipe is the winner.
PS. The __DOC_LOCAL and __DIR variables start with these magic variables below. These variables are a life saver and allow easy directory and file manipulation, they kind of setup a top-level context:
# Set magic variables for current file & dir
__DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
__FILE="${__DIR}/$(basename "${BASH_SOURCE[0]}")"
__SCRIPT="$(basename ${__FILE})"
__BASE="$(basename ${__FILE} .sh)"
__ROOT="$(cd "$(dirname "${__DIR}")" && pwd)"
if [ ! -t 1 ]; then
exec > /my/log/file 2>&1
fi
It should be noted that often the bottleneck is the terminal itself, try running your scripts with a "> /dev/null" to suppress output and verify the slow part is actually the script.
I would heartily agree. If your shell script grows beyond half a dozen commands or so, you're probably better off rewriting it in just about anything. Python, ruby, go (gorun), rust (cargo-script), or whatever else, doesn't really matter as long as it's not shell.
Anyway bash is still faster to use, and a lot of Unix services are based on it. The book is well written and have a great added value. Thank you for sharing!!
Ah, the luxury the modern breed of programmers enjoy today makes me feel jealous. In these days of Splunk and Document databases(returning jsons and xmls) its hard to understand why so many things in the past were the way they were.
Apart from DBMS interaction(Perl had DBI/x modules for that), pretty much every thing in the decade of 80's even upto late 2000's(tons of legacy systems) was so non standardized that people were literally parsing through log files and non standard data formats to store/exchange a lot of things, this was when the internet was growing crazy year over year, systems needed to be built and put in place. This means you really need to have first class facilities to manipulate text. You needed regexes baked neatly into the language. You needed qw, you needed ``, you needed while<FILEHANDLE>, you needed binary file handling features, you needed string manipulation facilities that could help you drill through any text file you could imagine, you needed powerful functional programming features, you needed OO etc etc. And you needed to get this done under tough deadlines on slow machines. Remember Java being cross platform compliant was one of the biggest selling point of the day. Perl had this before Java.
I personally worked on building a store using rcs and perl, that could store versioned config files, almost like a document data base. The parsing facilities required for the application we did demanded nothing short of a tool like Perl.
Python, and also Java growing rapidly once the data exchange formats were reduced to mark up languages and JSON. Suddenly you could with a library what most Perl programmers were doing using their language powers.
Also look at the Human Genome Project and Perl usage there.
Programmers take great pride in freeing accountants and ware house workers from drudgery. But seldom do we look at our work in the same way.
In the real world, most software work is done very similar to digging coal mines with shovels. Laborious manual hand typing jobs.
There are things you can do in line of perl that will take you 5 -10 min in python, but once program goes above one liners, difference is not that big.
And of course, if you learned vim instead of emacs, you would be even faster :))
f = open("ls|", "r)
f.read()
f.close() f = Popen('ls', stdout=PIPE).stdout
f.read()
f.close()
run('ls', stdout=PIPE).stdoutIn Perl you literally took the part of your shell pipeline, quoted it and opened it like any other file.
./output_generator | wc -l
becomes
open(SRC, "output_generator|");
open(WC, "|wc -l);
It's deprecated and does something quite different.
> If you ever dealt with much perl you know how much more pleasant and easy launching processes was in that language - like shell.
I'm sure it is, but you're missing my point.
> If you want to use that as evidence that I'm a garbage developer go ahead
Well that escalated quickly.
> open(SRC, "output_generator|");
> open(WC, "|wc -l);
>
> And you do whatever you want with those filehandles.
Again python does roughly the same, just with more overhead: a trailing pipe is an "stdout=PIPE", an input is a "input=<whatever>", and you access / forward stdout explicitly:
src = Popen('output_generator', stdout=PIPE)
wc = Popen(['wc', '-l'], input=src.stdout)
The python code to do piping ends up as longwinded and the plumbing of pipes ends up a massive headache so I wrote tidycmd to overcome that issue https://github.com/laurieodgers/tidycmd
cat something | grep "this" | cut -f 1 | sed -e 's/.../.../'
> cat something | grep "this" | cut -f 1 | sed -e 's/.../.../'
Literally none of this is actually useful if you're already in Python:
(
line.split('\t')[0].replace(…, …)
for line in open('something')
if 'this' in line
)
I can believe that if you're calling to external processes to perform operations which are pretty much trivial in the language.
This question is specific to pipes.
That's missing what I'm noting though, which is that you don't need pipes anywhere near a shell script if you can simply do more of your work in-language. Case in point being that the entire pipeline you cited as an issue has no reason to exist outside of a shell or shell script.
It doesn't feel like the language was designed for these tasks.
But oh, if you do set -o pipefail the grep will stop the whole pipeline when none of the lines matches "this". So you have to keep fiddling with ${PIPESTATUS[0]}. And none of pipefail or PIPESTATUS are really portable.
Not so simple after all.
If anything code that i used to write in Python at the past i write it in Bash nowadays, exactly because Bash has a better record when it comes to not breaking stuff. Though it helps that my Python use is also mostly scripts meant to run from the shell.
For these, being able to avoid the various $(echo... |sed) can be refreshing. Beside, the book makes for a nice repository of techniques.
And honestly I can't see any good argument for this patchwork approach to gluing things together. I guess some ops people might argue that you'd have to have ruby everywhere but the counter argument would be that we use docker images for everything and adding ruby as a dependency isn't any worse than all the insane dependency gymnastics it takes to get our node apps working.
And all of this applies equally for any language with a reasonable standard library (python, perl). I think people have weird feelings about using bash or make or whatever to accomplish things, like they are riding closer to the metal or that they are living some deeply pragmatic zen Unix philosophy, but mostly they are making an un-testable mess until it works once and then, if they are lucky, they don't have to touch it again.
Why I think people gravitate towards it is because languages such as python add too much pomp to launching a shell process. A language like perl is usually easier to use but everyone hates it now.