What Python is to Java, Shell is to Python. It speeds you up several times. I started using inline 'python -c' more often than the python repl now as it stores the command in shell history and it is then one fzf search away.
While neither Shell or SQL are perfect, there have been many ideas to improve them and for sure people can't wait for something new like oil shell to get production ready, getting the shell quoting hell right, or somebody fixing up SQL, bringing old ideas from Datalog and QUEL into it, fixing the goddamn NULL joins, etc.
But honestly, nothing else even comes close to this 10x productivity increase over the next best alternative. No, Thank you, I will not rewrite my 10 lines of sh into python to explode it into 50 lines of shuffling clunky objects around. I'll instead go and reread that man page how to write an if expression in bash again.
For those that haven't taken the time yet, I think this is a good place to start:
https://learnxinyminutes.com/docs/awk/
Of course, some people do very advanced things in awk and I absolutely agree that 1 hour of study isn't going to make you a ninja, but it's absolutely enough to learn the awk programming paradigm so that when the need arises you can quickly mobilize the solution you need.
For example: If you're quick to the draw, it can take less time to write an awk one liner to calculate the average of a column in a csv than it does to copy the csv into excel and highlight the column. It's a massive productivity booster.
There are people who have asked me scripting questions for over a decade, who will not read this for some reason.
It could be read in an hour, but not fully retained.
Printing based on another field, example gets UIDs >= 1000:
awk -F: '$3 >= 1000 {print $0}' /etc/passwd
Submitted yesterday:
Learn to use Awk with hundreds of examples
https://github.com/learnbyexample/Command-line-text-processi...
[1] https://www.gnu.org/software/gawk/manual/gawk.html
For the sarcasm impared among us, everything above this, but possibly including this sentence is sarcasm.
The quick one ~ true ~ fix was ! or #! without the 1024k copyright.
s-expression notation avoids the issue with (."contents")
MS Windows interpretation is much more terse & colorful.
Here's a nice example of something similar: https://drewdevault.com/dynlib
Scripted cholmskey grammar ( https://en.wikipedia.org/wiki/Universal_grammar ) to unleash the power of regular expressions.
Language aside, the ecosystem and culture do not afford enough in way of testing, dependency management, feature flags, static analysis, legibility, and so on. The reason people say to keep shell programs short is because of these problems, it needs to be possible to rewrite shell programs on a whim. At least then, you can A/B test and deploy at that scope.
Shameless plug coming, it this has been a pain point for me too. I found the issue with quotes (in most languages, but particularly in Bash et al) is that the same character is used to close the quote as is used to open it.m. So in my own shell I added support to use parentheses as quotes in addition to the single and double quotation ASCII symbols. This then allows you to nest quotation marks.
https://murex.rocks/docs/parser/brace-quote.html
You also don’t need to worry about quoting variables as variables are expanded to an argv[] item rather than expanded out to a command line and then any spaces converted into new argv[]s (or in layman’s terms, variables behave like you’d expect variables to behave).
I also write shell scripts, but I'm just curious what you would think about a comparison.
$X=q( foo() )
I agree that this qualified quotation is really underutilized.
Though Ruby makes it confusing AF because there are two quoting types for both strings and symbols, and they're different. (%Q %q %W %w %i) I can never remember which does which.... the letter choice feels really arbitrary.
https://elixir-lang.org/getting-started/sigils.html#strings-...
Elixir has sigils, which are useful for defining all kinds of literals easier, not just strings:
https://elixir-lang.org/getting-started/sigils.html#strings-...
You can also define your own. It's pretty great.
This also works up to a point where those GBs turn into hundreds of GBs, or even PBs, and a proper distributed setup can return results in seconds.
That's not to say I'd do everything in shell. Most stuff fits well into SQl, but when it comes to optimizing processing over TB or PB scale, you won't beat shell+massive hw orchestration.
Do you have an example of this? I didn’t even know you could make sql calls in scripts.
I don't do much sql in bash scripts but I do keep some wrapper scripts that let me run queries from stdin to databases in my environment
https://adamdrake.com/command-line-tools-can-be-235x-faster-...
PSQL="psql postgresql://$POSTGRES_USER:$POSTGRES_PASSWORD@$DATABASE_HOST:$DATABASE_PORT/$POSTGRES_DB -t -P pager=off -c "
OLD="CURRENT_DATE - INTERVAL '5 years'"
$PSQL "SELECT id from apt WHERE apt.created_on > $OLD order by apt.created_on asc;" | while
read -r id; do
if [[ $id != "" ]]; then
printf "\n\*\* Do something in the loop with id where newer than \"$OLD\" \*\*\*\n"
# ...
fi
doneDo you not have a ~/.python_history? The exact same search functions are available on the REPL. Ctrl-R, type your bit, bam.
But different strokes and all that.
* Shell scripts force you to think in a more scalable way (data streams)
* Shell scripts compose rich programs rather than simplistic functions
* Shells encourage you to program with a rich, extensible feature set (ad-hoc I/O redirection, files)
The only times I don’t like shell scripts are when dealing with regex and dealing with parallelism
What is used both in case/esac and globbing are "shell patterns." They are also found in variable pattern removal with ${X% and ${X#.
In "The Unix Programming Environment," Kernighan and Pike apologized for these close concepts that are easily mistaken for one another.
"Regular expressions are specified by giving special meaning to certain characters, just like the asterix, etc., used by the shell. There are a few more metacharacters, and, regrettably, differences in meanings." (page 102)
Bash does implement both patterns and regex, which means discerning their difference becomes even more critical. The POSIX shell is easier in memory for this reason, and others.
Wow, for me parallelism is one of the best features of a unix shell and I find it vastly superior to most other programming languages.
The comparison should be to perl or Ruby, both of which will fare better than Python for typical shell-type tasks.
It's a shell that is actually built for structured data, taking lessons learned from PowerShell and others.
Running `parallel --shellquote --shellquote --shellquote` and pasting in the line you want to quote thrice may alleviate some of the pain.
By no means ideal, though.
Not even that is necessary. Just use structured data formats like json. If you are consuming some API that is not json but still structured, use `rq` to convert it to json. Then use `jq` to slice and dice through the data.
dmenu + fzf + jq + curl is my bread and butter in shell scripts.
However, I still haven't managed to find a way to do a bunch of tasks concurrently. No, xargs and parallel don't cut it. Just give me an opinionated way to do this that is easily inspectable, loggable and debuggable. Currenly I hack together functions in a `((job_i++ < max_jobs)) || wait -n` spaghetti.
I would argue that once you pull in jq, you're no longer writing in "shell", you're writing in jq, which is a separate and different language. But that's precisely the point! Look at how effortless it is to (literally) shell out to a slew of other languages from shell.
The power of shell isn't in the scripting language itself, it's in how fluidly it lets you embed snippets of tr, sed, awk, jq, and whatever else you need.
And, critically, these languages callable from shell were not all there when shell was designed. The extension interface of spawning processes and communicating with arguments and pipes is just that powerful. That's where shell shines.
Perhaps some of the power emerges from that ambiguity, but it is quite difficult to implement.
This presentation sums up the woes of an implementor:
https://archive.fosdem.org/2018/schedule/event/code_parsing_...
It converts to/from the listed formats.
There is also `jc` (written in Python) with the added benefit that it converts output of many common unix utilities to json. So you would not need to parse `ip` for example.
This is a wrapper to jq that also supports yaml and other file formats.
Do you have an example of a script you wrote?
The scripts are similar to this one:
https://github.com/debxp/dmenu-scripts/blob/master/dmenu-kil...
I swear, there should be a HN rule against those. It pollutes every single Shell discussions, bringing nothing to them and making it hard for others do discuss the real topic.
It doesn't. You now have 4 numbers.
Five There's five numbers in this industry 0, 1, 3, 4, 5 and infinity
Wait, I'll come in again
The 7 being for design. If there are more than 7 boxes on the whiteboard, try again.
Unicode character can have more than 7 font boxes associated with one character box and still be a valid determinate character form.
log base 2 of 4 is only 16bits
This rule of ten lines or less is clearly meant as an illustrative guideline. Obviously if you have a shell script that has 11 lines, but does what it has to do reliably, nobody will be bothered.
The idea that the rule is trying to convey is "don't write long, complex programs in shell". Arguing about exact numbers or wording here is detracting from the topic at hand.
Unless, the language is APL, in which case, 10 lines is an operating system.
Handling binary data can also work in Bash, provided that you just use it as a glue for pipelines between other programs (e.g. feeding video data into ffmpeg).
One time, while working on some computer vision project, I had a need to hack up a video-capture-and-upload program for gathering training data during a certain time of day. It took me about 20 minutes and 50 lines of Bash to setup the whole thing, test it, and be sure it works.
Parsing text isn't anything fancy.
It's just knowing what the marker is for a word/item boundary.
For bash, that marker is defined in IFS
It is "opinion" based on debugging scripts made by people (which might be "you but few years ago") that don't know the full extent of death-traps that are put in the language. Or really writing anything more complex.
About only strong side of shell as a language is a pipe character. Everything else is less convenient at best, actively dangerous at worst.
Sure, "how to write something in a limited language" might be fun mental excercise but as someone sitting in ops space for good part of 15 years, it's just a burden.
Hell, I'd rather debug Perl script than Bash one...
Yeah, if it is few pipes and some minor post processing I'd use it too (pipe is the easiest way to do it out of all languages I've seen) but that's about it.
It is nice to write one-liners in cmdline but characteristic that make it nice there make it worse programming language. A bit like Perl in that matter
It's possible to use the wrong tool for the job in any language - including language choice itself.
Dismissing a programming language because it's not shell and dismissing shell because it's not a proramming language are the same thing - a bad idea if that's your only decision criteria.
If i need to run 11 commands in a row, suddenly i need to make sure new tooling is installed in my instance and/or ship a binary?
What if that 11 lines is setting up some networking? Now i need to go write a 400 line go program to use netlink to accomplish the same task? Or should I condense that to 80 lines of go to shell out the commands that replicate the 11 lines of simple bash?
There are plenty of reasons to do this, I have done it more than once. None of those reasons are "crossed an arbitrary magic number of 'lines of shell'".
/s
declare -A mydict( [lookma]=initalization )
mydict[foo]=bar
echo "${mydict[foo]}"
list=()
list+=(foo bar baz)
echo "${list[0]}"Try in Python to make a nested defaultdict you can access like the following.
d = <something>
d["a"]["b"]["c"] # --> 42
Edit: Dang it, I mean arbitrary depth.
c = defaultdict(lambda: 42)
b = defaultdict(lambda: c)
a = defaultdict(lambda: b)
a["a"]["b"]["c"] # --> 42Also d["a"] and d["a"]["b"] aren't 42.
d["a"]["b"]["c"] == (d["a"]["b"])["c"]
d = magicdict()
is42 = d["foo"]["bar"]["baz"]
# -> You can read any path and get a default if it doesn't exist.
d["hello"]["world"] = 420
# -> You can set any path and d will then contain { "hello": { "world": 420 }
I feel like we're maybe too in the weeds, I should have just said "now have two expressions in your lambda."
# One
a = d["a"]["b"]["c"]
# Two
a = d["a"]["b"]
b = a["c"]I have a hard time understanding any use case outside of such mocking.
Otherwise I'm not sure of a mainstream language that would let you do a.get(x).get(y) == 42 but a.get(x).get(y).get(z) == 42, unless you resorted to monkey patching the number type, as it implies 42.get(z) == 42, which seems.. silly
declare -A outer=(
[inner]="_inner"
)
declare -A _inner=(
[key]="value"
)
declare -n inner="${outer[inner]}"
echo "${inner[key]}"
# value
Bourne Shell Scripting is literally a bunch of weird backwards compatible hacks around the first command line prompt from 1970. The intent was to preserve the experience of a human at a command prompt, and add extra functionality for automation.
It's basically a high-powered user interface. It emphasizes what the operator wants for productivity, instead of the designer in her CS ivory tower of perfection. You can be insanely productive on a single line, or paste that line into a file for repeatability. So many programmers fail to grasp that programming adds considerations that the power user doesn't care about. The Shell abstracts away all that unnecessary stuff and just lets you get simple things done quickly.
* I like to the HGPPL data structures and convenient library for manipulating them (in my case this is Clojure which has a great core library). Bash has indexed and associative arrays.
* Libraries for common data formats are also used in a consistent way in the HGPPL. I don't have to remember a DSL for every data format - i.e. how to use jq when dealing with JSON. Similarly for YAML, XML, CSVs, I can also do templating for configuration files for nginx and so on. I've seen way too many naive attempts to piece together valid YAML from strings in bash to know its just not worth doing.
* I don't want to switch programming language from the main application and I find helps "break down silos" when everyone can read and contribute to some code. If a team is just sysadmins - sure, make bash the official language and stick to it.
* I can write scripts without repeating myself using namespaces and higher-order functions, which my choice of paradigm for abstractions, others write cleanly with classes. You can follow best practices, avoid the use of ENV vars, but that requires extra discipline and it is hard to enforce on other for the type of places where bash is used.
It's also important to learn your system's environment too. This is your "standard library", and it's why POSIX compatibility is important. You will feel shell is limited if you don't learn how to use the system utilities with shell (or if your target system has common utilities missing).
As an example of flexibility, you can use shell and system utilities in combination with CGI and a basic web server to send and receive text messages on an Android phone with termux. Similar to a KDE Connect or Apple's iMessage.
Which python can do realitively well, by using the `subprocess` module.
Here is an example including a https://porkmail.org/era/unix/award (useless use of cat) finding all title lines in README.md and uppercasing them with `tr`
import subprocess as sp
cat = sp.Popen(
["cat", "README.md"],
stdout=sp.PIPE,
)
grep = sp.Popen(
["grep", "#"],
stdin=cat.stdout,
stdout=sp.PIPE,
)
tr = sp.Popen(
["tr", "[:lower:]", "[:upper:]"],
stdin=grep.stdout,
stderr=sp.PIPE,
stdout=sp.PIPE,
)
out, err = tr.communicate()
print(out.decode("utf-8"), err.decode("utf-8"))
Yes, that is a convoluted example, but it illustrates the point I am trying to make. Everything outlined could probably done in a bash script, but I am pretty certain it would be much harder, and much more difficult to maintain, than doing this in python.
Bash is absolutely fine up to a point. And with enough effort, bash can do extremely complex things. But as soon as things get more complex than standard unix tools, I rather give up on the comfort of having specialiced syntax for pipes and filehandles, and write a few more lines handling those, if that means that I can do the more complex stuff easily using the rich module ecosystem of Python.
I am not sure I would agree. Sed fills this role quite nicely.
cat README.md | grep # | tr '[:lower:] [:upper:]' | sed 's/something/something_else/'
In python I can use `opt/argparse`, and use the error output from `re.compile` to do this.
Of course this is also possible in bash, but how easy is it to code in comparison, and how maintainable is the result?
If I wanted to parse cli args I would use case on the input to mux out the args. I personally prefer writing cli interfaces this way (when using a scripting language).
while test $# -gt 0; do
case "$1" in
-f|--flag) shift; FLAG="$1";;
esac
shift
donesed is the thing that handles shell regular expressions for shellscripts.
Doesn't sound so bad. A quick argument parser, a call out to grep or sed, pipe to curl, then to graphviz I guess (I don't really know much about image generation tools though), then compose the mail with a heredoc and run sendmail. Sounds like 10 to 15 lines for a quick and dirty solution.
Bash scripts have a nasty tendency to grow, sometimes in ways that are disproportional to the bit of extra functionality that is suddenly required. Very quickly, a small quick'n dirty solution can blow up to a compost-heap ... no less dirty, but now instead of a clean-wipe, I'd need a shovel to get through it.
I think my handle speaks for itself as to how much I like bash. But I have had the pleasure of getting handed over bash scripts, hundreds of lines long, with the error description being "it no longer works, could you have a look at it? and the original author both unreachable and apparently having string feelings against comments.
And in many of these cases, it took me less time to code a clean solution in Python or Go, than it took me to grok what the hell that script was actually doing.
And sure sure you can call any process from a language but the assumptions are different. No one wants to call a Java jar that has a dependency on the jq CLI app being available.
Shell is a tool that turns out to be pretty good for some things, particularly composing functionality out of other programs and also doing system configuration/tuning stuff to tailor an environment for other programs. It's also really handy for automating tasks you find yourself repeating.
Programming languages are a tool that are pretty good for other things - making new programs, tricky logic, making the most (or at least more than a shell script launching 1000s of new processes) efficient use of a computer.
Trying to replace one with the other is not really useful - they have different jobs. Learning to use them in conjunction on the other hand... there's a lot of power in that.
By comparison - javascript and html. They don't replace each other - yet they are both computer languages used in the same domain, and both have strengths and weaknesses. They have different jobs. And when you use them in conjunction you get something pretty darn powerful.
Clearly, it's for a different purpose, and there are some things that wouldn't work in a general-purpose language that isn't as focused on line-based string processing, but we are really happy with the things we took from bash.
This argument is essentially the same as "dynamic typing is just a different programming paradigm than static typing, and not intrinsically better or worse" - but to an even greater extent, because bash isn't really typed at all.
To those who think that static (and optional/gradual) typing brings strong benefits with little downsides over dynamic typing and becomes increasingly important as the size of a program increases, bash is simply unacceptable for any non-trivial program.
Other people (like yourself) that think that static typing isn't that important and "it's just a matter of preference" will be fine with an untyped language like bash.
Unfortunately, it's really hard to find concrete, clear evidence that one typing paradigm is better than the other, so we can't really make a good argument for one or the other using science.
However, I can say that you're conflating different traits of shell languages here. You say "dynamic scope, focus on line-oriented text, and pipelines" - but each of those are very different, and you're missing the most contested one (typing). Shell's untypedness is probably the biggest complaint about it, and the line-oriented text paradigm is really contentious, but most people don't care very much about the scoping, and lots of people like the pipelines feature.
A shell language that was statically-typed, with clear scoping rules, non-cryptic syntax, structured data, and pipelines would likely be popular and relatively non-controversial.
However the biggest issues I've had is that the code is really hard to test, error handling in shell isn't robust, and reusability with library type methods is not easy to organize or debug.
Those are deal breakers for me when it comes to building any kind of non trivial system.
There is something to be said in favor of the shell being always available, but Perl is almost always available. FreeBSD does not have it base of the base system, but OpenBSD does, and most Linux distros do, too.
But it is fun to connect a couple of simple commands via pipes and create something surprisingly complex. I don't do it all the time, but it happens.
- no standard unit testing
- how do you debug except with printlns? Fail.
- each line usually takes a minimum of 10 minutes to debug unless you've done bash scripting for... ten years
- basic constructs like the arg array are broken once you have special chars and spaces and want to pass those args to other commands. and UNICODE? Ha.
- standard library is nil, you're dependent on a hodgepodge of possibly installed programs
- there is no dependency resolution or auto-install of those programs or libraries or shell scripts. since it is so dependent on binary programs, that's a good thing, but also sucks for bash programmers
- horrid rules on type conversions, horrid syntax, space-significant rules
- as TFA shows, basic error checking and other conventions is horrid, yeah I want a crap 20 line header for everything
- effective bash is a bag of tricks. Bag of tricks programming is shit. You need to do ANYTHING in it for parsing, etc? Copy paste in functions is basically the solution.
- I'm not going to say interpreter errors are worse than C++ errors, but it's certainly not anything good.
Honestly since even effing JAVA added a hashbang ability, I no longer need bash.
Go ahead, write some bash autocompletion scripts in bash. Lord is that awful. Try writing something with a complex options / argument interface and detect/parse errors in the command line. Awful.
Bash is basically software engineering from the 1970s, oh yeah, except take away the word "engineering". Because the language is actively opposed to anything that "engineering" would entail.
With Trace. Which is talked about in TFA.
By the way nobody use exclusively bash. When i worked for a cloud provider, it was basically 30% python(ansible), 30% perl, 5 to 10% bash, and a bit of other languages depending on the client needs (mostly java, but also Julia and R).
Any example with this? The following works reasonably well for me.
args=(-a --b 'arg with space' "一 二 三")
someprog "${args[@]}"You think the complaints about rickety, unintuitive syntax are "socially founded"? I can't think of another language that has so many pointless syntax issues every time I revisit it. I haven't seen a line of Scheme in over a decade, and I'm still fairly sure I could write a simple if condition with less likelihood of getting it wrong than Bash.
I came at it from the other end, writing complex shell scripts for years because of the intuition that python would be overkill. But there was a moment when I realized how irrational this was: shell languages are enough of a garbage fire that Python was trivially the better choice for my scripts the minute flow control enters the picture.
Bash has dynamic scope with its local variables.
The standard POSIX language has only global variables: one pervasive scope.
Shell quoting though, Aieeee...
I find I have to shift gears quite substantially moving from shell or powershell to anything else...
"I'll just pipe the output of this function into.. oh, right"
So if you don't care about error cases everything is fine, but if you do, it gets ugly really fast. And that is the reason why other languages are probably be better suited if you want to write something bigger that 10 lines.
However, I have to admit, I don't follow that advice myself...
I guess you're referring to http://mywiki.wooledge.org/BashFAQ/105. Got recently hit by these as well.
My basic thesis is that Shell as a programming language---with it's dynamic scope, focus on line-oriented text, and pipelines---is simply a different programming paradigm than languages like Perl, Python, whatever.
Obviously, if your mental model is BASIC and you try to write Python, then you encounter lots of friction and it's easy for the latter to feel hacky, bad and ugly. To enjoy and program Python well, it's probably best to shift your mental model. The same goes for Shell.
What is the Shell paradigm? I would argue that it's line-oriented pipelines. There is a ton to unpack in that, but a huge example where I see friction is overuse of variables in scripts. Trying to stuff data inside variables, with shell's paucity of data types is a recipe for irritation. However, if you instead organize all your data in a format that's sympathetic to line-oriented processing on stdin-stdout, then shell will work with you instead of against.
/2cents