Given other tools are available is loading the question. Right now almost all Unix-like systems have Python, but which version 2.7 or 3.x? We still have boxes in our environment that only have 2.6. If they have 3.x, what is the x? Script portability in a diverse environment is a crucial issue for system scripts and even a lot of application support scripts.

Then there's flexibility. With bash you have a massive ecosystem of command line tools, including the ability to in-line code for tools like sed, awk and perl right there in your script. Python has a great standard library, but often the best way to do something in Python is using a third party module, so now you need to curate a module library across all your boxes. Also suppose your script needs to do things other than just process text, suppose I want to ssh onto a box and run a command? Is Paramiko installed, no? So now I need to invoke ssh in a sub-shell which is a PITA in Python compared to bash. Yes it's easier in later 3.x versions but now we're back to version soup again.

We have bash scripts running in our environment that are probably 20 years old, and if I write a new one now it will still work fine in 20 years time. Often it will be a quarter of the length of an equivalent Python script and much easier to understand and reason about.

Then again, there are many cases where a Python script makes a huge amount more sense, often run as part of a bash script.

I agree, this is probably not a good question. I think maybe it's indeed a case by case question and I have to master a lot of tools to make good judgement.

For now I'm teaching myself CL tools, and I feel if I need to pipe more than 5 times in one liners I'd rather use Python.

Awk and sed are generally superior for simple data transformations than python. Like cutting off specific lines, columns or replacing characters in files. They have pretty great performance in large files.

Once things get complicated though you should probably push whatever transformations you need to do to a database.

I would like to offer a counter-argument. I’ve been torrenting TV series for years, and I always have to do a mass rename of The files afterward to get Plex to process them properly and download the metadata. They’re always named something like “TV Show Name 2014.L337.H4XXX1080p-420.mkv”.

The other day I set out to use BASH to mass rename, and I had such trouble backslash-escaping the dot characters and whitespaces to prevent sed from interpreting them specially. Eventually I gave up and searched for the pythonic way to rename them, and it was as simple as a string replacement followed by a call to os.rename() inside a for-loop. It was a breath of fresh air to escape “command line Kung Fu” and fearing the thrashing from shell globbing.

To be fair, I got my start in using sed and awk as powertools in the BASH command pipeline, but I don’t miss them compared to a language with strong data types and simple built-in methods for handling complex manipulation. Python is built in to basically every Linux distro that has BASH, and for the sake of simple transformations, it offers a lot of succinct methods that work on either 2.7 or 3.x with no external packages.

Your example shows a very particular shortcoming in the unix command line programs. The mv command doesn't play nice with spaces in file names. I think python is a fine solution for your use case and wouldn't recommend against your code. Especially if you had to build it out for a lot of different patterns.

But if you're dealing with changing stuff in text files greater than 25 gigs I'd always recommend using the unix cmd line tool set for simple modifications if performance is a key concern. It works the same everywhere and can rip through big files. But I'm a Vim guy and I live in the shell so I'm surely quite biased lol.

There are plenty of pretty easy one liners though that can rename files depending on if you can define a pattern correctly though.

echo 'TV Show Name 2014.L337.H4XXX1080p-420.mkv' | awk -F . -v q="'" '{print q$0q" "q$1".mkv"q }' | xargs mv

find . -type f -name '*.mkv' -printf '%f' | awk -F . -v q="'" '{print q$0q" "q$1".mkv"q}' | xargs mv

You need zmv

Thanks! I hadn't heard of zmv before, and it fits my use case perfectly.

Thanks this is a good point. I did read some posts here that use sqlite for sort of middle storage in the pipelines.

When you are at the command line! (and you should be more) If you want to do quick analysis for log files, some data, then the command line is the best friend! no need to open any IDE, editor, no need to open excel files, just quick one-liners at worst multi liners, this is a super charge!

Sure, but you can invoke nearly all languages (and write one-liners) at the command line, e.g.:

python -c 'import sys; f=open(sys.argv[1]);print(len(f.readlines()))' .zshrc

But I wouldn't recommend python as a good one-liner language. So the question remains, what is awk particularly good at?

Think of Awk as being to traditional tab or whitespace separated Unix command output, as `jq` is to JSON command output. Extract a field, loop and iterate, template output, etc.

Now remember, this was popular when perl wasn't even something you could depend upon on all systems.

These days there's no reason you should even do anything terribly complicated in a nasty oneliner. Just save a nice function somewhere and call it with a concise alias.

Awk splits lines into numbered fields, so it’s really convenient for things like extracting and reordering columns from a csv file. For example, getting the second field of every line is:

  awk -F, '{ print $2 }'

I recently tried csvsql and it seemes to be a stronger tool to process csv files?

> But I wouldn't recommend python as a good one-liner language. So the question remains, what is awk particularly good at?

Processing regular (i.e. machine-, not human-generated) tabular text files line by line. If you already know Python there are probably few reasons to learn awk.

One of those could be working in constrained environments where you have awk but no Python. Awk is part of POSIX and part of Busybox, so there are almost no environments which have a working Python installed but no Awk, while the reverse is common enough (e.g. embedded systems). Apple plans to remove Python, Perl and Ruby from the default MacOS install in the next version, but probably not Awk (POSIX).

Good answers.

(And I hadn't realized Apple was planning on getting rid of Python, Perl, and Ruby from default installs! Wow.)

One-off jobs or jobs that you foresee won't be expanded.

For these jobs even an "import re" in Python is too much typing, compared to ubiquitous regular expressions in sed or awk. Awk is great at handling lines of delimited fields. Again in Python you would be writing your own boilerplate to read lines from files, splitting these lines, converting strings to numbers etc.

I don't know if I'm an "expert" but I only use awk and/or sed for simple shell scripts, when I'm munging around in a terminal, or for simple CLI tools I write for myself. If I'm writing code that I need to maintain, could grow, or needs to be contributed to in a team setting...I write it in a real programming language.