Well, that depends. Shell scripts can be commented, and they can be built progressively and interactively by building a pipeline. That's a great choice for a one-off task, and in my experience much faster than many other approaches.
It also works well for tasks close to the system. For example, our users are able to download large archives of data, and we keep over 99% of such downloads indefinitely. We delete downloads > 100GB after they're more than 6 months old.
With a shell script run by cron that's achieved with find + rm + curl|jq (to tell the API the download is deleted).
I've gotten surprisingly far using just curl/cat/cut/sed/awk/wc & friends. When I need to build on top of that, I go write a real program, but the UNIX-fu tells me what I need to build on top of.
I recommend mk (original make replacement for Plan 9, available for different OSes through Plan9Port [0,1]). It has a bit more uniform syntax, and can check that the dependency graph is wellfounded.
mk is not there yet.
and even if that is the case, I'm still better being a vim user at a system that only has vi and ed, then any other editor's user.
I have so many `zgrep ... | awk ... | sort` scripts that I legitimately couldn't tell you what they do anymore, just what the correct output is.
I really like the feeling of proving to myself that I know how to use my command line well, but in most cases I end up spending longer trying to remember the differences between OS X built-in `sed` and the `sed` I know and blah. Lots of wasted time.
I've started keeping a "scratch" git repo with scripts for one-offs, so the next time I say "oh I want to go through this CSV and run X on each line with Y" I can look at my old code and replace the necessary parts.
echo x |\
# Comment here
cat | # Or like this
cat
> 8 LOC scraper that those without bash knowledge will be unable to parse.
And someone without knowledge of Node.js will be unable to parse the 50-line JavaScript monstrosity.
> And the 8 LOC bash script is not much of a time savings as this post demonstrates; you still have to check each function output manually to find which data to parse / handle edge cases.
It only took that long because he detailed every step as a tutorial. If you have basic literacy of the shell, it's no time at all.
To refute you, I decided to solve it myself, using only the "hint" at the beginning of the article to use ?action=raw to work with the source wikitext (I had not yet read the rest of the article; I had not seen his solution).
It took me literally 2 minutes: (Setting in url='https://en.wikipedia.org/wiki/List_of_Olympic_medalists_in_j... for readability on HN)
[2016-08-15 17:52] curl -s ${url}
[2016-08-15 17:52] curl -s ${url}|grep flagIOCmedalist
[2016-08-15 17:52] curl -s ${url}|grep -o '{{flagIOCmedalist.*}}'
[2016-08-15 17:53] curl -s ${url}|grep -o '{{flagIOCmedalist.*}}'|cut -d'|' -f2|sed 's/[[\]]//g'
[2016-08-15 17:53] curl -s ${url}|grep -o '{{flagIOCmedalist.*}}'|cut -d'|' -f2
[2016-08-15 17:54] curl -s ${url}|grep -o '{{flagIOCmedalist.*}}'|cut -d'|' -f2|sed 's/\(\[\|\]\)//g'
[2016-08-15 17:54] curl -s ${url}|grep -o '{{flagIOCmedalist.*}}'|cut -d'|' -f2|sed 's/\(\[\|\]\)//g'|sort |uniq -c
[2016-08-15 17:54] curl -s ${url}|grep -o '{{flagIOCmedalist.*}}'|cut -d'|' -f2|sed 's/\(\[\|\]\)//g'|sort |uniq -c|sort -n
Sure, for something that might need to run repeatedly, it probably doesn't handle future edge cases that might arise. But it's not production, it's a one-off exploring the data script. Not every one-off program you write needs to be production quality.
And being worried about "those without bash knowledge"... don't be afraid to use your operating system!
That said, this line of solutions has a big edge case: it relies on the editors of Wikipedia being consistent and formatting each row as one line in the source. If I were to solve this totally on my own, choosing to ignore the "hint", and work with the rendered HTML, and just do most of it with a nokogiri one-liner.
Knowing that there's an ?action=something to get just the page HTML without the navigation and such. I spent about 3 minutes finding the "render" action (documented here: https://www.mediawiki.org/wiki/Manual:Parameters_to_index.ph... ). Then anther 3-ish minutes poking around the DOM in my browser to get an idea of what I'm working with, and visually inspecting the layout of the article; each cell with a medalist has two links, the first to the medalist, and the second to the country. Then it took me a whopping 5 minutes to hammer out the rest of the one-liner.
(Similarly, url='https://en.wikipedia.org/wiki/List_of_Olympic_medalists_in_j... for HN readability)
[2016-08-15 18:14] curl -s ${url}
[2016-08-15 18:15] curl -s ${url}|nokogiri -e '$_.css("table.wikitable td a:first-child").each{|a| puts a.text}'
[2016-08-15 18:16] curl -s ${url}|nokogiri -e '$_.css("table.wikitable td a:first-child").each{|a| puts a.text}'|grep -vE '^[0-9]{4} '
[2016-08-15 18:17] curl -s ${url}|nokogiri -e '$_.css("table.wikitable td a:first-child").each{|a| puts a.text.sub!("\n", " ")}'
[2016-08-15 18:18] curl -s ${url}|nokogiri -e '$_.css("table.wikitable td a:first-child").each{|a| puts a.text.sub("\n", " ")}'
[2016-08-15 18:18] curl -s ${url}|nokogiri -e '$_.css("table.wikitable td a:first-child").each{|a| puts a.text}'|grep -vE -e '^[0-9]{4} ' -e '^details$'
[2016-08-15 18:19] curl -s ${url}|nokogiri -e '$_.css("table.wikitable td a:first-child").each{|a| puts a.text}'|grep -vE -e '^[0-9]{4} ' -e '^details$'|sort |uniq -c|sort -n
For a grand total of 11 minutes; generously.
My point is: Don't be afraid to play around with your tools and data! Have fun! Not everything needs to be production quality! And being written primarily in bash doesn't necessarily mean that it isn't production quality!
Make -I dir
Make --include-dir=dir
anyone who uses single letter options on a script should be punished.
single letters are for one time typing.
This is a bad attitude to have for working with data processing, where QA is necessary and the accuracy of the output is important. A 50 LOC scraper with comments and explicitly-defined inputs and output from functions is far preferable to a 8 LOC scraper that those without bash knowledge will be unable to parse.
And the 8 LOC bash script is not much of a time savings as this post demonstrates; you still have to check each function output manually to find which data to parse / handle edge cases.