awk 'BEGIN {FS=","} /cake/ {print $2, $3}' *.csv > cakes.csv
Unsurprisingly I disagree with the post's description of awk being an "advanced command".I guess there's no categorization of "advanced" vs "beginner" that will satisfy every audience but I consider awk an advanced tool. About 20 years ago, I wrote some AWK tips and cheatsheet back on USENET and today I would have to refer to that post to write basic awk commands.
The thing about awk is that it's a compact programming language with variables and conditionals and that's a step-change in complexity for many users.
I think it can be pretty advanced, for me awk is one of those tools where I still feel like I need to write a paragraph of comments to explain 1 line of code.
For example: https://github.com/nickjj/invoice/blob/75660dce5a29ceb4e47a6...
Keep in mind I don't really "know" awk. I cobbled that together from a few examples. It will convert times formatted like "2h 30m", "150m" or "2:30" into 2.50. There's a bunch of examples in the test file.
NOTE: I wrote that script 2.5 years ago and I know there's questionable patterns in other areas of the script that's not highlighted like using a bunch of separate echo calls instead of a heredoc.
Shell scripting is really fun and efficient. I use it all the time for a variety of things.
awk -F, '/cake/ {print $2, $3}' *.csv > cakes.csv
does the same thing: -F sets the field separator.I still remember when I started working as a sysadmin at 19, the greybeard UNIX guy taught me how to vim, and he told me awk was as important as knowing vim, pointing to some huge AWK manual he had on the shelf, one of those with the animal in the cover.
This was 15 years ago, I know vim, but awk still eludes me.
For a quick introduction, see [1] [2]
[0] https://github.com/learnbyexample/learn_gnuawk
- I understood where a lot of the stuff in Perl (and, to a lesser degree, Ruby) came from, especially Perl's implicit loop mode.
- I never wanted to write a program in awk again. Ruby (or whatever your preferred scripting language is) is better every time: awk is not much less work and Ruby generalizes more.
Awk for scripting is a bit like shell for scripting. It's easy to extend a script into something which is uncomfortably expensive to rewrite when you inevitably want to do something it's not well suited to. Awk still has uses, but it's in things like portable shell scripts which need more text manipulation power.
But awk is usually not a good choice for processing CSV files.
Your program will fail if any of the fields include embedded commas (which is often the case).
$ ls | grep '.csv$' | xargs cat | grep 'cake' | cut -d, -f2,3
cake
cake
cake
$ grep cake *.csv | cut -d, -f2,3
bar.csv:cake
foo.csv:cake
quux.csv:cake
`grep -h` will do the trick thoughThis is a tool used to accomplish a thing, and Unix tools can be used to accomplish things in many different ways. This is like complaining about, I don't know, using a metric-labeled screwdriver on an imperial-measured screw that still gets the job done exactly as needed. Cut it out.
This is super ignorant. You risk stripping the screw and getting yourself into a frustrating screw extraction job.
Just because you lived, doesn't mean it was safe.
I would say the ignorant one of us is the one who isn't taking into account the nature of the job being done? What makes you think that a) my task is life or death and also b) I'm not equipped to figure out how dangerous it is?
To pull out the camera. I'd like to spread Unix tools to everyone, or at least "more people than use them now." I understand though, for a lot of people, they'd rather gatekeep.
You just shouldn't use an "almost"-fitting screwdriver or wrench.
I'm not trying to "gatekeep." I'm not trying to talk about the shell metaphorically. I'm literally talking about screws.
> The primary cause of this discrepancy was that one piece of ground software supplied by Lockheed Martin produced results in a United States customary unit, contrary to its Software Interface Specification (SIS), while a second system, supplied by NASA, expected those results to be in SI units, in accordance with the SIS.
https://en.wikipedia.org/wiki/Mars_Climate_Orbiter#Cause_of_...
It's hardly gatekeeping to try to teach people better ways to do things.
The reason many people in IT can't think outside of this idea is that they are (subconsciously) reluctant to open the gates for others. As a teacher in IT I quite literally deal with this every day.
The OP even included the reasons why their version is preferable.
You really see that as complaining and gatekeeping? An 'utterly and completely necessary "correction."'?
Anti-patterns are bad, because it usually means that sample command might work in this case, but not in other environments or other use-cases. Someone who is seeing these commands first time, has no idea about that. And this post is meant for beginners.
I uselessly use "cat" ALL THE TIME, because the Unix pipe metaphor is absolutely intuitively excellent. I know that I'm starting at the beginning with pipe and shoving data through, and I have a good idea about each step.
It is not a matter of taste and it is not a matter of metric versus imperial screwdrivers. Someone will copy this code and it will end up being an attack vector where it will have consequences.
I imagine you're rolling your eyes and have flipped the bozo bit but please bear with me.
Think of the teachable moment this presents! The author of the original piece goes back and annotates their original answer along the lines of, "you might solve it this way but there are some gotchas with it - let me show you what could go wrong."
As an industry we absolutely need to circle back with improvements so that those who come after us can build on a more solid foundation.
So filenames can contain valid commands delimited with semi-colons that, if not quoted properly, can be unexpectedly run alongside your intended pipeline (say if you're doing the usual and unsafe "for csv in *.csv; do cat $csv; done").
I wish I could've laid my hands on the excellent HN thread from some years back that opened my eyes to this vector, but I'm hopeful someone else will mention it so I can add it to my bash notes file. :P
Parsing CSV with simple text-oriented tools is bad of an idea as parsing HTML with regexps.
https://csvkit.readthedocs.io/en/latest/
https://github.com/BurntSushi/xsv
Heck, even sqlite has some stuff to help with processing CSV files: https://www.sqlite.org/csv.html