That's just, like, your opinion man...
> people who still do use complex solutions built from cat, head, cut, etc, and who know what they're doing, will typically either write a shell script or use awk
No, I still use cat, head, cut etc., because it's easier to see at each step what is happening and incrementally add to that, because they're literally everywhere, because it's quick, and because I like it. Not for major projects, granted, but why would I need to write a python file for something that takes a single line of piped commands?
Ugh.
Why is it not valid today?
> One-liners which pipe text between four or five different commands are the equivalent of hand-soldered boards or bitwise arithmetic.
Why is that bad? Don't deploy a supercomputer to do what a hand soldered board can. Keep it simple.
I don't see how you equate piping commands to bitwise arithmetic, but bitwise arithmetic is easy anyway.
> but of no practical utility
Says you. Just because you don't find something useful doesn't mean nobody else finds it useful.
Turning one liner piped commands to a program in what you might consider a "proper programming language" usually ends up turning a declarative program into something prodecural. Not that that's necessarily a bad thing. Just saying.
Use the right tool for the job. In some cases (not all) the shell is indeed the right tool.
I get a feeling you don't understand the Unix philosophy. Read The Art of Unix Programming by Eric S. Raymond. Go learn bitwise arithmetic.
The uneducated play with pictures. Educated people read and write :)
Have a great day (or night, depending on your timezone — night here).
These kind of comments aren't useful, and come off smug to me.
Why doesn't he understand something, what it the point in reading/learning something you think relevant? Why would they invest in your suggestions if you don't provide any reasons what is missing?
Also:
> The uneducated play with pictures. Educated people read and write :)
This sounds insulting to me (name-calling), and passive-aggressively so when combined with "Have a great day".
I don't understand - isn't bash shell procedural?
I think you actually confirmed my point when you said 'no sysadmin has time or interest'. I didn't say that no one uses shell anymore. I said that there was no practical justification for doing so. Lots of people still use it and don't have any interest in finding a better solution. These people are choosing, for their own reasons, to double down on an obsolete skillset. Good for them, I suppose, but I don't think that demand for their niche is going to be around for very long.
I also pointed out that sysadmins who know what they're doing use awk. Based on your mentioning awk it seems like you agree with this.
To move stuff to that cattle and to maintain cloud environments, to prepare k8s clusters and manage those. To create docker images or to maintain servers that are running k8s.
You still have AWS console and Azure CLI or GCP console and configurations that need to be search/replaced. You still need to parse logs to understand what is happening in multiple environments, which I would say makes it even more important to be good with unix-fu because you have to find that one broken cow to put it out of misery.
Those command line tools are still useful and used.
I specifically named awk as a way of solving these type of problems that has a similar applicability and maturity to a scripting language. My real objection is to long chains of cut, head, cat, grep, and so on. I also think that utilities like jq and xargs, which were invented exactly to allow people to do things in shell one-liners which are trivial in a proper language, are worse than useless.
I didn't say that no one should use awk, or no one should use the command line.
In fact I didn't say that no one should use pipes with head, cut and grep either. I just said that most reasons to use them were no longer valid.
Lots of shell script in that part of the code world.
This is false. Every line they write has potential security implications. If any pattern can become a string injection vulnerability, it will. Even most real programmers do not understand shell scripting.
This discussion is all moot because UN*X is an obsolete misconception made by programmers who are too dumb to understand the difference between and significance of AST manipulation and string concatenation. It's not hard to understand what I'm talking about. Look how much ad-hoc, non-reusable trivia you have to learn to pass stuff around between find, ls, and xargs. Whenever you would do `x = f(); g(x)` in Python, you will spend 10 minutes figuring out if some given shell script is doing the equivalent the correct, safe, secure way.
I just realized today that every time I see some crap like x,abc%%20%%20def,y x,abc\x20\x20def,y I am actually depressed because I already know all the played out meta of such systems - it's a bunch of half working junk full of pointless vulns that only exist because between 0 and 10 of the said "experts at their craft" in the world actually bother to program this crap correctly. And I literaly have to squint to figure out where the bug is (there's always the bug in such code).
If the UN*X shell was replaced by Python, sysadmins would have no trouble adapting. Python is a terrible language, (plus this discussion is convoluted by the fact that Python's gimped from bending over to work well with UN*X) but still better than UN*X shell in every way.
Why use many word when few word do trick?
Shell scripts are very compact for what they do, and generally as clear as a similarly quick-and-dirty solution in another language. Ease of learning and use comes from having used the programs you are running in your script. E. g. my small script piping stuff from nmcli to fzf so I can choose a wifi network would be much more difficult to write in Python: I’d need to find a Python library for interacting with NetworkManager, and a library for interactively fuzzy searching in a list, read the docs, spend a while setting up a venv to run it, … I don’t have time for any of that.
xargs, in particular, is not difficult to invoke once you’ve done it once or twice, and does a lot. Apart from just being a loop, it also parallelizes execution, and can ask for confirmation from the user for each invocation. Implementing either of these features will take you more than 2-4 chars you need to use it with xargs.
I think the perfect very high level language is closer to shell than to python for example. The power of Tcl/Tk (still it has some big weaknesses) or Rebol/Red is something that I admire.
The following statement is probably controversial: shell is more akin to Lisp for human beings. I dabbled at Scheme, but it is harder for me to grasp than Shell, but in the end they are more similar than not.
I hold a candle for Oil shell for example.
The only downside of shell scripting is that it isn't trivially portable to Windows (or even macOS because of the differences between GNU and BSD tools), so it often makes sense to create big Python scripts that do more than "one thing right". If the whole world would run on UNIX, shell scripting would make much more sense.
The only downside? Let's add that no major *NIX shell that I'm aware of has any good way to modularize code while enforcing encapsulation of state.
At my previous job, we had a rule that any shell script longer than about a page of code had to be replaced with a Python script ASAP. That was a good rule, IMO, because once you've exceeded a certain size, a shell script starts getting brittle and hard to work with. I don't know if 1 page of code is the threshold size or not, but it seems like as good a cutoff as any.
Shell is portable in a way nothing else is, same reason people use Excel instead of code or PHP instead of literally anything.
Secondly the presence of all these utilities on a machine is far from guaranteed - expecting Python to be present is no more or less likely.
Why do you get to declare what is acceptable (using UNIX tools) or what's now obsolete (SSHing into a host)? It's rather presumptuous to believe you can speak for everyone. Perhaps you only ever deploy code using k8s and never need to use a command line but that doesn't mean we all do. There are many reasons one would use these tools and approaches (fun, small scale, investigating problems, anything).
For what it's worth, my bosses all the way up to CEO of a 50k person company were all pretty pleased with the working practices I used (medieval things like strace(1), perf(1) and sort(1)) to save thousands of servers' worth of CPU. I'm not advocating for editing code live over SSH but there are still plenty of legitimate reasons to SSH into a server and use UNIX tools.
Since you're clearly trolling I'll leave it here.
Of course lots of people do it because they enjoy it or are used to it, and for them cost and related considerations don't matter. The same way that lots of people ride horses.
So you don't need to SSH to debug, you'd just re-deploy another container or something.
However, there's an <<extremely>> long tail of places, equivalent to mom and pop stores, where SSH will still be in use for a long, long time. I include antiquated banks or government in "mom and pop stores" for the purposes of this discussion :-p
I've still never seen a place where this was true.
At some point you always run into a situation where you need to see what's happening inside the container in production at a level of detail that isn't available except by directly connecting. So you attach a sidecar with SSH, or you deploy a debug version of your container that runs SSH, and then you SSH into your container and see what's happening.
(Again – an honest question! I was never formally instructed in these matters.)
On macOS, there's still only Python 2.7 pre-installed (Xcode then goes ahead and also installs an equally outdated Python 3 version). On Windows the Python executable is an empty stub which opens the app store.
I wish the situation would be better.
1. The version of bash is even more out of date (Dating back to 2007 on current macOS releases).
2. #!/bin/sh Could give you any one of a number of slightly different shell implementations.
3. The differences between Python versions are clearly documented, and, while painful, possible to work around. The differences between shell versions can be far more subtle and more difficult to discover.
4. You not only need to deal with the shell version, but the version of every tool you use as well.
As opposed to trying to directly launch /bin/sh on Windows? :-p
The practical utility has just been demonstrated in this particular article? Historical? I think that Unix shell are like crocodiles - outliving the dinosaurs and lurking in the murky water the unsuspecting sysadmin to come close enough to fix the script that ain't broken.
How? The author is doing something which could be done much more easily and elegantly in a programming language.
Strongly disagree. The understanding of the OS, the data, and how to checkmate the problem with minimal effort is timeless.
Programming languages are relatively ephemeral compared to POSIX utilities.
Invest in knowledge of the enduring.
Some questions:
1) Where do you draw the line, and move away from shell to a "real" language. If I just want want to view a directory, surely I use ls, right? What about if I want to remove all leading whitespace from a text file? This can be done with a short but fairly opaque awk one-liner. Probably takes way more time to write and run the Python equivalent, but I don't know Python so well, so maybe it's also a one-liner.
2) What's the optimal "real language" for replacing shell? Python? Perl? Raku?
2a) xargs automatically parallelizes programs. How can this be done efficiently (meaning I don't have to write much additional code) in your proposed "real language"?
By using xargs :)
Or more likely gnu parallel. But seriously, in so many cases using GNU parallel to parallelize a process is the quickest and easiest way to approach the problem, and I use it all the time. If I need to process 10k+ images in a folder, rather than try to parallelize the process in my python/C script I'll write the fastest possible single threaded script that takes its input args as command line arguments and then uses gnu parallel to distribute the workload. The added advantage is that I can distribute this work on a cluster of machines with only a few changes to GNU Parallel's command line arguments.
Shell is indeed very old and it's time for a replacement, but it's not there yet.
Oilshell might get there eventually or at least spark interest in this area.
Solved. https://github.com/ngs-lang/ngs
I'm the author. Frustrated with exactly this situation I created Next Generation Shell. It's a "proper programming language" on one hand but domain-specific for "DevOps"y scripting on another. So sane syntax, data structures, error handling, multiple dispatch on one hand but also syntax for running external programs, pipes and redirects.
You are welcome!
You cherry-picked the one thing that shell script is somewhat better at than other languages (not even really needed for this task). Meanwhile the article uses shell for both making HTTP requests and mangling JSON data, both of which are easy in all modern languages, and extremely painful in shell.
> (ii) Expect the output of every program to become the input to another, as yet unknown, program. Don't clutter output with extraneous information. Avoid stringently columnar or binary input formats. Don't insist on interactive input.
https://homepage.cs.uri.edu/~thenry/resources/unix_art/ch01s...
But what if it's not? It's perfectly easy to write code to do the same thing in a programming language without invoking any other executables, let alone piping the output of one into the input of another. So the claim that this functionality is what makes Unix and shell scripting effective is vacuous. They are good at doing something which would not be necessary if one was not using them.
If the article was about using ls to just list files, and I had said "actually you should use Python's os.listdir() and filter the results by whatever" you would be right.
For most simple problems it's correct to use a simple tool. For the overwhelming majority of complex problems you should use a well-understood, well-designed, common general-purpose tool.
If this isn't the case, stay away.
Note that people who still do use complex solutions built from cat, head, cut, etc, and who know what they're doing, will typically either write a shell script (which won't be structured particularly differently from the equivalent Python or whatever) or will rely heavily on awk (itself a full-featured programming language, no easier to learn than any other scripting language), or both.
One-liners which pipe text between four or five different commands are the equivalent of hand-soldered boards or bitwise arithmetic. Interesting to learn about for historical reasons but of no practical utility.
The use of things like xargs and jq in this solution, difficult to invoke Unix utilities for doing things that are trivial in any reasonable language, makes this even more clear.