I can sympathize with the author w.r.t wanting to use a single language you like for everything. However, after decades I've found this to be untenable. There are languages that are just simply better for one-off scripting (Perl, Python), and languages that aren't (anything compiled). Trying to bolt an interpreter onto a compiled language from the outside seems like a lot of work for questionable gain.
One reason is deployment. Writing code in python/node/etc... implies the ability of the production environment to bootstrap a rather complicated installation tree for the elaborate runtimes required by the code and all its dependencies. And so there are elaborate tools (npm, venv, Docker, etc...) that have grown up around those requirements.
Compiled languages (and Go in particular shines here) spit out a near-dependency-free[1] binary you can drop on the target without fuss.
I deal with this in my day job pretty routinely. Chromebooks have an old python and limited ability to pull down dependencies for quick test runs. Static test binaries make things a lot easier.
[1] Though there are shared libraries and runtime frameworks there too. You can't deploy a Gnome 3 app with the same freedom you can a TCP daemon, obviously.
But I think for python you could also deploy a binary with pyinstaller.
I think it's more accurate to say "static binaries" instead of "compiled languages." The same headaches exist with dynamically linked, compiled binaries (and sometimes they're worse since you don't have a dependency manager, unless you add one)
> Static test binaries make things a lot easier I think this really depends on your target environment and how much control you have over it. If you're in a Ruby or Python shop, for example, all your servers already have the stack installed. If you're targeting end user devices, those can have a huge mess of different configs to account for
It's nothing like that with Python or Node. The rule there is that you get something working locally and then spend a while reverse engineering a pip/venv recipe or manifest or whatever to make it work somewhere else. It's decidedly non-trivial.
I could imagine a lifetime of collecting scripting macros/libs in lisp to be as good or better.
However, the reason Bash is so prolific amongst Sys Admins such as myself is the fact that they are portable and reliable to use across Debian, Arch or RHEL based distributions.
You don't have to import extra libraries, ensure that you are running the proper python environment, or be certain that pip is properly installed and configured for whatever extra source code beyond what is included out of the box.
Bash is the most consistent code you can write to perform any task you need when you have to work with Linux.
Python is (at least in the CPython implementation) compiled, to python byte code which runs on the python virtual machine.
Its not compiled to native code. (Unless you use one of the compilers which do compile it to native code, though they tend to support only a subset of python.)
Once you use it to manage complex data structures and flow, you are simply wasting time because you will have to rewrite it in Python or Go.
Just like your post does! :)
Until you try to use a newer feature or try the script in a Mac or BSD or any older bash.
SH code is completely portable, but bash itself can have quite a few novel features. Don’t get me wrong - I’m happy the language is dynamic and still growing. But it can make things awkward when trying to use a script from a newer system on an older server (and the author has been “clever”).
Not exactly.
https://stackoverflow.com/questions/11376975/is-there-a-mini...
Installing python or nodejs into your distro will inflate it by at least 30mb, quite crucial when dealing with containers
That requires a level of bookkeeping which is helpful for large programs and a nuicense for small programs.
I do have to add that Python more than any other language I've used results in working the first try that it's not surprising.
actually amount of reasoning, which program requires to perform in run time is close to interpretion.
Not false, but perhaps in need of better definition. The term script has often denoted a trivial set of commands run by $interpreter.
"Scripting languages" have been seen as being in contrast to C, C++, Pascal, Java, SmallTalk, &c. The scripting languages remove from the user the need:
-a- to think about an extensive type system,
-b- to compile the logic, and
-c- to build for a specific architecture.
Yes, and if the thing I'm trying to do has a small input, it will only be done once, etc. I will often just pipe `grep` to `sort` or whatever, because it's less typing, it's generally clearer to a wider range of people, etc.
But on larger inputs, or even things like doing a single pattern inversion mixed with a pattern match, I like awk.
I've made this point before, but I still find it hilarious. For more than a decade, awk was dead. Like, dead dead. There was nothing you could do in awk that wasn't cleaner and simpler and vastly more extensible in perl. And, yes, perl was faster than gawk, just like gawk is faster than shell pipelines.
Then python got big, people decided that they didn't want to use perl for big projects[1], and so perl went out of vogue and got dropped even for the stuff it did (and continues to do) really well. Then a new generation came along having never learned perl, and...
... have apparently rediscovered awk?
[1] Also the perl 5 tree stagnated[2] as all the stakeholders wandered off into the weeds to think about some new language. They're all still out there, AFAIK.
[2] Around 2000-2005, perl was The Language to be seen writing your new stuff in, so e.g. bioinformatics landed there and not elsewhere. But by 2015, the TensorFlow people wouldn't be caught dead writing perl.
Learning awk a bit more deeply in recent years has been good too. I can write one liners that do more. I shipped a full awk script once, for something unimportant, but I would never do that again. For serious text munging these days I'd rather write a Rust program.
By the way, I think this is why Perl lost to Python on larger scripting and programming projects - it's just easier to write (albeit harder to read, to antagoinze the Python lovers out there).
Also the classic sysadmin role which used to lean on Perl heavily sort of evolved with rise of The Cloud and automation tools like Chef, Puppet, and Ansible took over in that 2005-2015 time frame.
EDIT: I decided to ask GPT-4 to translate the gawk script to Perl. I make zero claims that this is ideal (as stated, I don't know Perl at all), but it _does_ produce the same output, but slightly slower than the gawk script.
$ hyperfine "perl -lane '\$ips{\$F[0]}++; END {print \"\$ips{\$_} \$_\" for (sort {\$ips{\$b} <=> \$ips{\$a}} keys %ips)[0..9]}' fake_log.txt"
Benchmark 1: perl -lane '$ips{$F[0]}++; END {print "$ips{$_} $_" for (sort {$ips{$b} <=> $ips{$a}} keys %ips)[0..9]}' fake_log.txt
Time (mean ± σ): 1.499 s ± 0.006 s [User: 1.447 s, System: 0.050 s]
Range (min … max): 1.490 s … 1.507 s 10 runs perl -e 'for $i (<>) { $i =~ s/ .*//; push @list, $i; }; print(sort(@list));'And in a world where perl is a natural choice for these tasks, awk doesn't have a niche. Because at the end of the day awk is simply an inferior language.
Which is the bit I find funny: we threw out and forgot about a great tool, and now we think that the ancestral toy it replaced is a good idea again.
AWK, on the other hand, is simplicity itself. Sure, it misses a few things, but for searching through log files or db dumps it's an excellent tool. And it's fast enough. If you really need much more speed, there are other tools, but I would rather rewrite it in C than try perl again.
Original: 3.294s
w/ LC_ALL=C: 1.055s
w/ larger sort buffer `-S5%`: 0.780s
Your gawk: 1.772s
+ LC_ALL=C: 1.772sI did find one other interesting difference between BSD and GNU tools - BSD sort defaults to 90% for its buffer, GNU sort defaults to 1024 KiB.
Combining all of these (and using GNU uniq - it was also faster), I was able to get down to 463 msec on the M1 Air:
$ hyperfine "export LC_ALL=C; gcut -d' ' -f1 fake_log.txt | gsort -S5% | guniq -c | gsort -rn -S5% | head"
Benchmark 1: export LC_ALL=C; gcut -d' ' -f1 fake_log.txt | gsort -S5% | guniq -c | gsort -rn -S5% | head
Time (mean ± σ): 463.4 ms ± 3.3 ms [User: 965.5 ms, System: 93.3 ms]
Range (min … max): 459.9 ms … 469.8 ms 10 runs
$ hyperfine "export LC_ALL=C; cut -d' ' -f1 fake_log.txt | sort -S5% | uniq -c | sort -rn -S5% | head"
Benchmark 1: export LC_ALL=C; cut -d' ' -f1 fake_log.txt | sort -S5% | uniq -c | sort -rn -S5% | head
Time (mean ± σ): 1.504 s ± 0.318 s [User: 2.833 s, System: 0.474 s]
Range (min … max): 0.942 s … 1.937 s 10 runs
$ hyperfine "export LC_ALL=C; cut -d' ' -f1 fake_log.txt | sort | uniq -c | sort -rn | head"
Benchmark 1: export LC_ALL=C; cut -d' ' -f1 fake_log.txt | sort | uniq -c | sort -rn | head
Time (mean ± σ): 3.847 s ± 0.093 s [User: 4.165 s, System: 0.613 s]
Range (min … max): 3.591 s … 3.919 s 10 runs
Syscalls aren't free, nor are disk reads, so if you have the RAM to support slurping the entire file at once, it's sometimes faster.
The script package looks really cool and I'll definitely try it out, cause honestly even though I do a lot of bash scripting it's super painful for anything but something super simple.
{
PROCINFO["sorted_in"] = "@val_num_desc"
top_ips[$1]++
}
END {
counter = 0
for (i in top_ips) {
if (counter++ < 10) {
print top_ips[i], i
}
}
}
At least half of what people complain about with shell scripts can be solved by using ShellCheck [0], and understanding what it's asking you to do. I disagree with the common opinion of "anything beyond a few lines should be a Python script instead." If you're careful with variable scoping and error handling, bash is perfectly functional for many uses.
I don't think this is true. Before I learned Go, I could follow along most Go programs pretty well, and learning Go well enough to get started took less than an hour. Every attempt I've made to learn more Awk, I've bounced off.
Whereas if you know awk and bash, then they can be incredibly useful in a pinch. It doesn’t knock how powerful they are. I think it is worth learning. But if something needs to be maintained then there is an argument for Python/Go/whatever.
“Loaded guns are perfectly functional for juggling, just be careful with the trigger and you won’t shoot yourself in the foot!”
You are technically correct but why bother with being careful when you could just avoid writing bash?