It's funny how many things that seem kind of incredible for a pure bash solution, looking back over the past 21 years doing this drivel. This year is the first I had to learn how to export all of the shell's variables and function definitions for a forked clone to run asynchronous background processing of the main thread. Bash! Bash of all things. Probably 10 seconds in any other language, but here is bash kind of hopping alongside. Callbacks, reflective programming, there's always a really awkward way in bash to do what's happening in the popular languages.
My go to example of “Bash can do whaaat?!” is the source for xip.io. A custom DNS server written in a handful of lines of Bash!
https://github.com/basecamp/xip-pdns/blob/master/bin/xip-pdn...
In my fevered state, I wrote some arithmetic functions to do the equivalent of `expr $a + $b` and `expr $a \* $b` without any fork-exec to external commands, just to prove to myself that it could be done. I think I abused things like $IFS, for loops, recursive function calls, and case statements to make an arbitrary-precision decimal adder function.
Even by looking at these examples, you see it has less verbosity like "then" and "do", you can reference arguments as $argv instead of cryptic $@ and exit status code as $status instead of $? which is confusing with $! and the likes.
https://blog.codeship.com/lets-talk-about-shell-scripting/
https://fishshell.com/docs/current/tutorial.html
Shell is such an integral part of admins and programmers workflow yet I find it hard to believe this field has been so slow at improving. Even the fish site jokingly states "Finally, a command line shell for the 90s" implying the others are even older.
You can simply import aliases as is and in some cases, you may be able to even simplify parts of your config.
Personally, it was easy for me as I'm administering the servers and it was all just installing fish on every servers. (some dozens.)
Absolutely, yes. Because long story short, bash is TriedAndTrue® technology. Stuff that works.
It takes a bit of dedication to be mastered at a decent level but it pays off immensely. It's so ubiquitous it is one of those tools that you can learn once, use for the rest of your life, and use in a lot of contexts.
It's so widespread that it can bring you very far with very little.
All this being said, as someone who used to write code for a living (and now works as a System Engineer, using the bash shell everyday) I must say that if you do not do input verification (according to the language of your choice) and something goes wrong then it's your fault.
More like "better the devil you know" (if you can even say that much).
Bash doesn't scale. Every shop larger than 1 has been burned by bash gotchas. Use a real scripting language and shell out to the commands and builtins when necessary.
At my shop bash is strictly disallowed in production environments and we're all better off for it.
Thanks mate, I had a good laugh.
I wouldn't expect bash to scale anyway. That's not what it's meant for. It's meant for system administration task automation.
On a more serious note ...
In many occasions, the performance you get depends on how you tackle the problem you have, though. Even using bash and the tools from the unix toolbox, sometimes you can gain significant improvements on how you manage your data.
Anecdotal: I cannot remember the details, but I remember that rearranging the order of sorting and searching and removing duplicates (sort, sort -u, grep, uniq mainly) I saw a significant speedup.
Anecdotal (2): I cut the execution time of a night-running job from hours to minutes (tens of minutes, to be honest - but still less than an hour) just by slicing the size of a problem into smaller parts and by handling each slice in parallel (the machine had 48 cpus, but the problem was being solved "sequentially" on one cpu alone). I wrote some 30-50 lines of python, just to implement parallelism control: the rest of the problem was still handled with bash script. Partial results were reassembled at the end. Bash has coprocesses, so I might have handled that in bash as well, but python was more handy at the time (meh, i just wanted to optimize that problem).
What I am trying to say is that sometimes the "scaling" you get is justified by the size of the problem, sometimes it's not.
Again, it really depends on how you handle your data.
Having a number of filters chained via pipes is really efficient, for example, when compared with looping over an array and executing some python/perl/ruby one-liners every time.
Bash is not at all meant for "system administration task automation", the very idea is ludicrous. It has to be the most singularly ill-purposed invention ever to be applied towards task automation.
Not to mention, for example: https://julialang.org/blog/2012/03/shelling-out-sucks
In the case of shells, not benefitting from the extra decades of experience. There are some horrible things that you can do with Bash just by mistyping a single character or using the wrong type of quotes or failing to appreciate how something will expand in some edge case. This is not a desirable property for an environment that people use after being paged at 4am, while their employer is losing $XXX,000/minute because something critical is down, and where the half-asleep operator is one short command away from deleting the universe without so much as a confirmation prompt.
https://github.com/lihaoyi/Ammonite
Many times previously I've starting really quickly testing/prototyping an idea in bash, which eventually grew large and unweildy. Ammonite appears like it might allow quick hacking, but also the option to easily transition your scripts into scala proper while leveraging the good bits of Java.
> Bash has arrays and a safe mode, which may make it just about acceptable under safe coding practices, when used correctly.
> Fish is easier to use correctly, but lacks a safe mode.
> Prototyping in fish is therefore a good idea, provided that you know how to translate correctly from fish to bash.
> Fish is easier to use correctly, but lacks a safe mode. Prototyping in fish is therefore a good idea, provided that you know how to translate correctly from fish to bash.
Which leads right into 5: that domain is unsafe-by-design and failure-tolerant. Which is phenomenally useful and efficient for one-off tasks, and phenomenally dangerous for something you allow others to inject behavior into.[1]
There are a fair number of things which bash could do better, and probably should. But many of the gotchas like "if you don't quote it, it'll expand into multiple arguments" are features when you want to throw things around efficiently. It wouldn't be as efficient for plugging so many disparate tools together if it weren't such an efficient footgun.
[1]: If you need anything even remotely fault-resistant, `set -euo pipefail` and use `shellcheck` (and the rest of this resource, it is indeed great) and then consider using a "normal" programming language instead. But for behavior you fully control it's not too bad if you're careful.
Stories! C'mon, it was a long Monday :-)
I also started capturing most tasks as bash functions or aliases in my `~/.bashrc` file. This doubles as both a handy reference for me to look at later to remember how to do something, and automating simple things.
sed in particular is really amazing in what it can do, very simply. (Note, depending on your system, you might want to get GNU sed, for "extended" regex support.)
Those pages do a great job breaking down the basic behavior and explaining pitfalls. They're skim-able, and pretty easily searchable after than, so you can come back and remind yourself of the thing you learned last week.
As much as I think some of the new generation of tools are pretty awesome for getting stuff done, some of it honestly feels like a step backwards because you're relying on 3rd party services or require a new server(s) to be provisioned to host your management servers; some of which require extra man power just to keep running. And as much as Bash gets a bad rep for hidden traps, I'd take that any day over Terraforms "I'm not really a programming language but I like to pretend to be" markup, HCL.
But that's the nature of the game now; it's less about hacking stuff together and more about using preformed Lego bricks to build the same approximation.
I concur with the same statement. Newer domain specific syntaxes and tools are increasing the dependencies and cognitive load to get things done. One would have to remember the yaml syntax, and version specific keywords for Kubernetes, Docker, Ansible and everything in the container world just to deploy a statically linked Go binary. That's where we've come to now. It is worth pondering whether we are increasingly moving towards "give me the lego bricks, and I don't care to know what it specifically does" attitude, and if it's good or bad.
1/ HCL not being a complete language is a burden when you're writing build scripts that need to target multiple environments (eg dev, UAT, staging, live) as it means you have to use the `count` syntax to include/exclude resources. Also I've frequently run into issues where output will expect a variable or resource to exist even when it's inside a ternary operator where the condition isn't met (isn't the point of an `if` condition that you don't evaluate all blocks of code in the structure?).
2/ The reason HCL isn't a "complete language" isn't because "complete languages" are a liability (clearly that's nonsense); it's because Hashicorp wanted something that met the middle ground between JSON and scripting so that all ranges of technical expertese were comfortable; and they wanted a format that could still be compiled back down to JSON. Personally I feel the monster they created is the worst of both worlds but I do think the goal they were trying to achieve is an honorable one.
3/ I was never trying to make the point that I've built more reliable systems in Bash. However once you start writing pretty complex Terraform projects (some of my build scripts are a several thousand lines of code; inc modules) you really do start to bang your head with Terraform - not all of it because of HCL though. Don't get me wrong, Terraform is definitely the best tool we have at the moment for deploying to "the cloud"; but that doesn't mean it isn't still bloody annoying to work with at times. A few bugs I've run into in the last month:
3a/ Error message wasn't being passed from AWS to the user so infra would fail to build but without explanation. A few hours of debugging and we discovered what resource was failing and why. This was a bug in Terraform so issue raised on Github.
3b/ Workspace name pushed the resource name tag over it's character limit. We did get an error message here but it was inaccurate. Thankfully this is an issue we've run into before so it was pretty quick to resolve.
3c/ The aforementioned problem of all code getting evaluated inside both conditions of a ternary operator
3d/ If your AWS token expires before the Terraform apply is complete you're left with incomplete infrastructure and no valid state file to rollback. This is particularly annoying if your builds do certificate management or ELK; both of which can take 15 to 20 minutes just on those resources alone.
3e/ You don't get line numbers nor even file names when errors are raised which makes debugging through large projects painful.
4/ I'm not about to sing the praises of Bash as a modern deployment tool as it can be a complete nightmare to work with if you're not proficient in it's hidden traps. But to answer your question; yes I have built reliable systems in Bash. It's a tool that has been around for decades, people can and have built some pretty reliable stuff in it and I personally consider myself a veteran. (for what it's worth I have nearly 3 decades of dev skills and have used a lot of other languages we might now consider "dangerous" - some of which are still used in production). As an aside note; it's the decades of years of experience in Bash and other, much lower level, languages that inspired me to write my own $SHELL and scripting language. So I tend to use that more than Bash these days.
5/ I didn't proof read this so apologies if some parts didn't make any sense. It's a long post and in replying to you I'm interrupting fixing one of those massive Terraform projects I described earlier.
Just use Perl, Python or Ruby. One or all of them is installed on every machine you are likely to use.
If you install just one gem, bundler, you can get dependency management in a one-file script. Checkout Bundler Inline:
https://github.com/bundler/bundler/blob/master/lib/bundler/i...
I'd really like to use Python more often, but most of the time it only complicates things with those verbose syscalls.
EDIT: Seems I've only tried too basic things - it works thanks to %automagic, and seemingly %man is a thing. Other option is to use %%sh or %%bash, but that's a bit verbose.
Nope. I really need to be able to test commands on the terminal and then copy/paste them without any modifications. I won't even try to get used to that stupid syntax.
https://www.youtube.com/watch?v=pb3k0sGKrjQ&t=457s
'Take bash seriously'
> Perl is a interpreted language optimized for scanning arbitrary text files, extracting information from those text files, and printing reports based on that information. It's also a good language for many system management tasks. The language is intended to be practical (easy to use, efficient, complete) rather than beautiful (tiny, elegant, minimal). It combines (in the author's opinion, anyway) some of the best features of C, sed, awk, and sh, so people familiar with those languages should have little difficulty with it. (Language historians will also note some vestiges of csh, Pascal, and even BASIC-PLUS.) Expression syntax corresponds quite closely to C expression syntax. If you have a problem that would ordinarily use sed or awk or sh, but it exceeds their capabilities or must run a little faster, and you don't want to write the silly thing in C, then perl may be for you. There are also translators to turn your sed and awk scripts into perl scripts.
-- Larry Wall, perl 1.0 release announcement, comp.sources.unix v13i001, 1988-01-02.
some_command \
| grep -E 'some.*regex$' \
| sed -e 's/erase_text//g' \
| sed -e 's/erase_more//g' \
| awk '{ print $2 }' \
| cut -d ':' -f 1
If the script is complex enough and I'm willing to take a dependency on Ruby, that is usually where I turn. Where that line is does change from time to time. Ruby makes it super trivial to call shell commands and process them easily as well, which makes it easy to integrate into existing scripting [3].
Hopefully that answered your question :-)
[1] http://technosophos.com/2009/05/21/perl-pie-if-you-only-lear...
sed -e 's/erase_text//g; s/erase_more//g'
some_command | gawk ' /some.*regex$/ {gsub(erase_text,"");gsub(erase_more,"");split($2,a,":"); print (length(a[1]) ? a[1] : "STRING ERROR")}'
I like the pattern. Copied it to my notes to try next time I am grepping through logs.
I usually just string a few 'grep -v ignore_this' with pipes but then it ignores the whole line. But I can see how erasing some parts of the line would be very helpful sometimes.
some_command |
grep -E 'some.*regex$' |
sed -e 's/erase_text//g' |
sed -e 's/erase_more//g' |
awk '{ print $2 }' |
cut -d ':' -f 1If you're interested, here's an article I wrote recently that attempts to explain the method in the madness: https://medium.com/p/the-weird-wondrous-world-of-bash-arrays...
1. This article contains excellent advice and should be starred for later retrieval.
2. Having basic scripting skills will make you a way better programmer. Many times I've done huge refactors and needle-in-hay-stack searches using only shell commands.
3. Shell is the universal language.
4. Bash isn't that bad once you get used to it (seriously. I'll grant you tho that arrays are still nasty ;-) ).
5. Bash is not that dangerous if you follow best practices. Don't be lazy!
6. You will not regret getting really good at shell script. You'll have to take my word for it now because you don't know what you're missing.