News for hackers. Stuff that matters.
Re:Cheating at a company group activity using Unix tools by l0b0 on Monday December 13, 2021 @05:21AM (#29538322)

  ls | grep '.csv$' | xargs cat | grep 'cake' | cut -d, -f2,3 > cakes.csv
That's quite a few antipatterns in one go. Unless you have a bajillion files the `xargs` is unnecessary, the `cat` and `ls` are unnecessary (and `ls` in shell scripts is a whole class of antipatterns by itself). You might want to use something like this instead:

  grep cake *.csv | cut -d, -f2,3 > cakes.csv
Re:Cheating at a company group activity using Unix tools by gbrown_ on Monday December 13, 2021 @06:08AM (#29538548)
I certainly agree with your points, though the original task is a textbook example of where awk shines.

    awk 'BEGIN {FS=","} /cake/ {print $2, $3}' *.csv > cakes.csv
Unsurprisingly I disagree with the post's description of awk being an "advanced command".
Re:Cheating at a company group activity using Unix tools by jasode on Monday December 13, 2021 @07:40AM (#29539018)
>I disagree with the post's description of awk being an "advanced command"

I guess there's no categorization of "advanced" vs "beginner" that will satisfy every audience but I consider awk an advanced tool. About 20 years ago, I wrote some AWK tips and cheatsheet back on USENET and today I would have to refer to that post to write basic awk commands.

The thing about awk is that it's a compact programming language with variables and conditionals and that's a step-change in complexity for many users.

Re:Cheating at a company group activity using Unix tools by nickjj on Monday December 13, 2021 @06:53AM (#29538767)
> Unsurprisingly I disagree with the post's description of awk being an "advanced command".

I think it can be pretty advanced, for me awk is one of those tools where I still feel like I need to write a paragraph of comments to explain 1 line of code.

For example: https://github.com/nickjj/invoice/blob/75660dce5a29ceb4e47a6...

Keep in mind I don't really "know" awk. I cobbled that together from a few examples. It will convert times formatted like "2h 30m", "150m" or "2:30" into 2.50. There's a bunch of examples in the test file.

NOTE: I wrote that script 2.5 years ago and I know there's questionable patterns in other areas of the script that's not highlighted like using a bunch of separate echo calls instead of a heredoc.

Shell scripting is really fun and efficient. I use it all the time for a variety of things.

Re:Cheating at a company group activity using Unix tools by nicce on Monday December 13, 2021 @12:01PM (#29541454)
awk is Turing-complete, so if that is not enough to describe ”advanced” tool, then I don’t know what it is.
Re:Cheating at a company group activity using Unix tools by cross on Monday December 13, 2021 @01:44PM (#29542813)
So is sed.
Re:Cheating at a company group activity using Unix tools by vlovich123 on Monday December 13, 2021 @05:49PM (#29545714)
I’d classify basic search/replace functionality of sed as “basic” - you see such scripts frequently enough in shell scripts. Same for awk and “extract fields 1 and 3”. Anything beyond that quickly escalates into intermediate and advanced usages. Often you can find an SO post for lots of intermediate usages (where you can just include it as a link saying “I’m trying to do X”) but advanced usages that require knowledge of that are best avoided unless you are investing in sed/awk expertise in your team. Usually choosing something like Python, Ruby, Node or even Perl can provide better value because all of those languages can solve the same problem and solve other problems those tools can’t.
Re:Cheating at a company group activity using Unix tools by yesenadam on Monday December 13, 2021 @06:32AM (#29538660)

  awk -F, '/cake/ {print $2, $3}' *.csv > cakes.csv
does the same thing: -F sets the field separator.
Re:Cheating at a company group activity using Unix tools by 1_player on Monday December 13, 2021 @06:17AM (#29538589)
awk is one of those underrated tools I always wanted to learn.

I still remember when I started working as a sysadmin at 19, the greybeard UNIX guy taught me how to vim, and he told me awk was as important as knowing vim, pointing to some huge AWK manual he had on the shelf, one of those with the animal in the cover.

This was 15 years ago, I know vim, but awk still eludes me.

Re:Cheating at a company group activity using Unix tools by asicsp on Monday December 13, 2021 @06:49AM (#29538753)
I wrote a book on GNU awk one-liners with hundreds of examples and exercises [0]. Free to read online.

For a quick introduction, see [1] [2]

[0] https://github.com/learnbyexample/learn_gnuawk

[1] https://backreference.org/2010/02/10/idiomatic-awk/

[2] https://earthly.dev/blog/awk-examples/

Re:Cheating at a company group activity using Unix tools by yesenadam on Monday December 13, 2021 @06:28AM (#29538639)
Read The AWK Programming Language, a joy to read, one of the finest docs ever written, I reckon. You'll be up and running in minutes.

https://archive.org/details/pdfy-MgN0H1joIoDVoIC7

Re:Cheating at a company group activity using Unix tools by smcameron on Monday December 13, 2021 @09:25AM (#29539821)
If you know C and regex, that's 85% to 95% of awk. The basic layout of an awk program is a sequence of regex/action pairs, where the regexes are between slashes, and the actions are between curly braces. (There's also the special regex values of BEGIN and END, executed prior to the first line of input and after all input is consumed, respectively). The action part has a syntax very much like C, with the special values $n (where n is an integer) representing field number n for a any input line. There are a bunch of special functions it defines that you can call, like substr(), or split(), and good old printf. Awk processes the input stream one line at a time, matching each line against the series of regexes, and if it matches a regex, it executes the action paired with that regex, with all the $n variables filled in with field values from that input line. And that's basically enough for 99% of what awk is used for on a day to day basis.
Re:Cheating at a company group activity using Unix tools by barrkel on Monday December 13, 2021 @10:48AM (#29540631)
I wrote one script in awk and learned a lot:

- I understood where a lot of the stuff in Perl (and, to a lesser degree, Ruby) came from, especially Perl's implicit loop mode.

- I never wanted to write a program in awk again. Ruby (or whatever your preferred scripting language is) is better every time: awk is not much less work and Ruby generalizes more.

Awk for scripting is a bit like shell for scripting. It's easy to extend a script into something which is uncomfortably expensive to rewrite when you inevitably want to do something it's not well suited to. Awk still has uses, but it's in things like portable shell scripts which need more text manipulation power.

Re:Cheating at a company group activity using Unix tools by gorgoiler on Monday December 13, 2021 @09:22AM (#29539782)
My thoughts exactly, though with “-F,” to quickly set the field separator.
Re:Cheating at a company group activity using Unix tools by devnull255 on Monday December 13, 2021 @08:51AM (#29539548)
Awk is advanced in the sense that it is a programming language by itself. Years ago I had to migrate an old version of an Informix database to a newer version. The old version did not have tools to export the database as DDL and DML statements. So I had to create a tool myself to do it. The system did not have perl installed so I had to use awk. It worked nicely enough.
Re:Cheating at a company group activity using Unix tools by coolgeek on Wednesday December 15, 2021 @10:10AM (#29566961)
I love awk. I've been using it for over 25 years.

But awk is usually not a good choice for processing CSV files.

Your program will fail if any of the fields include embedded commas (which is often the case).

Re:Cheating at a company group activity using Unix tools by darrenf on Monday December 13, 2021 @05:28AM (#29538363)
Almost, but not quite :)

    $ ls | grep '.csv$' | xargs cat | grep 'cake' | cut -d, -f2,3
    cake
    cake
    cake
    $ grep cake *.csv | cut -d, -f2,3
    bar.csv:cake
    foo.csv:cake
    quux.csv:cake
`grep -h` will do the trick though
Re:Cheating at a company group activity using Unix tools by darrenf on Monday December 13, 2021 @12:11PM (#29541623)
I'm surprised, hours later, that no-one has pointed out my error here! So I'll do it myself - upon revisiting it, I realised my input files weren't CSVs with the number of fields required for `cut` to do the intended thing. Had I made them so, the `-h` isn't required after all. My bad :)
Re:Cheating at a company group activity using Unix tools by jrm4 on Monday December 13, 2021 @11:21AM (#29540980)
I really wish people would mostly stop doing this type of utterly and completely necessary "correction." It really misses the point, and exemplifies nearly the exact opposite of the point and value the original article is expressing.

This is a tool used to accomplish a thing, and Unix tools can be used to accomplish things in many different ways. This is like complaining about, I don't know, using a metric-labeled screwdriver on an imperial-measured screw that still gets the job done exactly as needed. Cut it out.

Re:Cheating at a company group activity using Unix tools by DarylZero on Monday December 13, 2021 @11:50AM (#29541311)
> using a metric-labeled screwdriver on an imperial-measured screw that still gets the job done exactly as needed

This is super ignorant. You risk stripping the screw and getting yourself into a frustrating screw extraction job.

Just because you lived, doesn't mean it was safe.

Re:Cheating at a company group activity using Unix tools by jrm4 on Monday December 13, 2021 @12:57PM (#29542210)
Sure, let's beat up this metaphor.

I would say the ignorant one of us is the one who isn't taking into account the nature of the job being done? What makes you think that a) my task is life or death and also b) I'm not equipped to figure out how dangerous it is?

To pull out the camera. I'd like to spread Unix tools to everyone, or at least "more people than use them now." I understand though, for a lot of people, they'd rather gatekeep.

Re:Cheating at a company group activity using Unix tools by DarylZero on Monday December 13, 2021 @02:05PM (#29543062)
It doesn't matter whether it's life or death. If you end up having to extract that screw it's going to be a big headache.

You just shouldn't use an "almost"-fitting screwdriver or wrench.

I'm not trying to "gatekeep." I'm not trying to talk about the shell metaphorically. I'm literally talking about screws.

Re:Cheating at a company group activity using Unix tools by carapace on Monday December 13, 2021 @02:47PM (#29543621)
You picked a bad metaphor, eh?

> The primary cause of this discrepancy was that one piece of ground software supplied by Lockheed Martin produced results in a United States customary unit, contrary to its Software Interface Specification (SIS), while a second system, supplied by NASA, expected those results to be in SI units, in accordance with the SIS.

https://en.wikipedia.org/wiki/Mars_Climate_Orbiter#Cause_of_...

It's hardly gatekeeping to try to teach people better ways to do things.

Re:Cheating at a company group activity using Unix tools by philwelch on Tuesday December 14, 2021 @04:50AM (#29549955)
Comparing the failure of a multi million dollar space probe to a stripped screw head seems like an even worse analogy to me.
Re:Cheating at a company group activity using Unix tools by carapace on Tuesday December 14, 2021 @11:00AM (#29553009)
The moral of the story is "Don't mix metric and imperial." Using that as an example of a trivial thing that shouldn't be corrected is foolish and ignorant. It's practically inviting Murphy.
Re:Cheating at a company group activity using Unix tools by philwelch on Wednesday December 15, 2021 @12:46PM (#29569189)
The moral of the story is “don’t mix units”. It would have been the same problem if the units were inches and feet or meters and kilometers as it is with feet and meters.
Re:Cheating at a company group activity using Unix tools by jrm4 on Monday December 13, 2021 @04:10PM (#29544693)
No, it's a good one. What y'all mistakenly assumed was the idea of "every computer task is a life or death one" instead of considering the idea that there can be different levels of care for different levels of danger.

The reason many people in IT can't think outside of this idea is that they are (subconsciously) reluctant to open the gates for others. As a teacher in IT I quite literally deal with this every day.

Re:Cheating at a company group activity using Unix tools by carapace on Monday December 13, 2021 @05:42PM (#29545646)
I really don't think that teaching people better ways to do things is exclusionary.

The OP even included the reasons why their version is preferable.

You really see that as complaining and gatekeeping? An 'utterly and completely necessary "correction."'?

Re:Cheating at a company group activity using Unix tools by howdydoo on Monday December 13, 2021 @01:13PM (#29542435)
Slightly off topic, but is this something people worry about in the real world? I have about 9 random screwdrivers, and when I need one I just try them all until one seems to fit. And if it doesn't, I... stop, instead of stripping the screw. I couldn't tell you if a single one of them was metric or imperial
Re:Cheating at a company group activity using Unix tools by BenjiWiebe on Tuesday December 14, 2021 @01:29PM (#29555287)
I work on small engines. Their carburetors usually are held together with JIS screws, which do look like Phillips. You can tell if someone took it apart with Phillips, though, as the screw is (basically anyways) visibly damaged and the screwdriver doesn't seat as well.
Re:Cheating at a company group activity using Unix tools by DarylZero on Monday December 13, 2021 @02:09PM (#29543116)
There is no such thing as metric or imperial screw heads. There are various ISO/ANSI standards.
Re:Cheating at a company group activity using Unix tools by jrm4 on Monday December 13, 2021 @04:08PM (#29544661)
:) That's the other thing about my example, it was just a doofy little thing I made up.
Re:Cheating at a company group activity using Unix tools by nicce on Monday December 13, 2021 @12:07PM (#29541565)
It might not miss the point, if the command sample is giving more complicated demonstration than it actually is, almost like doing artificially long command to demonstrate unix-magic, to get more audience.

Anti-patterns are bad, because it usually means that sample command might work in this case, but not in other environments or other use-cases. Someone who is seeing these commands first time, has no idea about that. And this post is meant for beginners.

Re:Cheating at a company group activity using Unix tools by jrm4 on Monday December 13, 2021 @12:54PM (#29542169)
I don't think it's more complicated. Most of the time I see people doing "corrections" on the so-called proper way to use Unix tools, they're favoring conciseness (sure, whatever) over intuitiveness.

I uselessly use "cat" ALL THE TIME, because the Unix pipe metaphor is absolutely intuitively excellent. I know that I'm starting at the beginning with pipe and shoving data through, and I have a good idea about each step.

Re:Cheating at a company group activity using Unix tools by bonkabonka on Monday December 13, 2021 @01:22PM (#29542553)
The original example MUST be corrected for the same reason that folks who post naive code snippets adding SQL strings together with user input must be corrected.

It is not a matter of taste and it is not a matter of metric versus imperial screwdrivers. Someone will copy this code and it will end up being an attack vector where it will have consequences.

I imagine you're rolling your eyes and have flipped the bozo bit but please bear with me.

Think of the teachable moment this presents! The author of the original piece goes back and annotates their original answer along the lines of, "you might solve it this way but there are some gotchas with it - let me show you what could go wrong."

As an industry we absolutely need to circle back with improvements so that those who come after us can build on a more solid foundation.

Re:Cheating at a company group activity using Unix tools by vorador on Monday December 13, 2021 @01:39PM (#29542767)
Genuinely curious, what's so bad about the original code that it adds an attack vector?
Re:Cheating at a company group activity using Unix tools by bonkabonka on Monday December 13, 2021 @05:58PM (#29545805)
To guard against malicious filenames I use `find ... -print0 | xargs -r0...` since posix disallows null bytes (and forward slashes) in filenames. The `-r` flag on xargs means it doesn't execute its command if find matches nothing.

So filenames can contain valid commands delimited with semi-colons that, if not quoted properly, can be unexpectedly run alongside your intended pipeline (say if you're doing the usual and unsafe "for csv in *.csv; do cat $csv; done").

I wish I could've laid my hands on the excellent HN thread from some years back that opened my eyes to this vector, but I'm hopeful someone else will mention it so I can add it to my bash notes file. :P

Re:Cheating at a company group activity using Unix tools by cuu508 on Monday December 13, 2021 @11:40AM (#29541215)
This is pointing out you can simplify the drill+pliers+plunger+screwdriver contraption to just screwdriver.
Re:Cheating at a company group activity using Unix tools by XorNot on Monday December 13, 2021 @07:04AM (#29538810)
I'd go further and say don't parse CSV with plaintext tools because it's barely a plaintext format. Use a CSV library and save yourself heartache when someone drops a quoted string in somewhere.
Re:Cheating at a company group activity using Unix tools by oblio on Monday December 13, 2021 @09:07AM (#29539661)
I don't understand why you're being downvoted.

Parsing CSV with simple text-oriented tools is bad of an idea as parsing HTML with regexps.

Re:Cheating at a company group activity using Unix tools by devenvdev on Monday December 13, 2021 @09:20AM (#29539764)
Most likely because it contradicts most people's experience, some CSVs can't be parsed with cli tools, but most of them can be, and it's much easier than writing code that does the same. So what the parent commenter says is true, just not pragmatic.
Re:Cheating at a company group activity using Unix tools by oblio on Monday December 13, 2021 @10:35AM (#29540490)
It is pragmatic with a minor tweak.

https://csvkit.readthedocs.io/en/latest/

https://github.com/BurntSushi/xsv

Heck, even sqlite has some stuff to help with processing CSV files: https://www.sqlite.org/csv.html

Re:Cheating at a company group activity using Unix tools by philwelch on Tuesday December 14, 2021 @04:59AM (#29550006)
csvkit and SQLite were always too slow whenever I tried to use them for this sort of thing. I don’t remember if I tried xsv. I do think I ended up converting most of my CSVs to TSV, or exporting them as TSV in the first place.
Re:Cheating at a company group activity using Unix tools by ryanianian on Monday December 13, 2021 @10:14AM (#29540271)
This isn't parsing CSV, it's generating it. That's not nearly as fraught.
Re:Cheating at a company group activity using Unix tools by philwelch on Tuesday December 14, 2021 @04:56AM (#29549988)
In my experience doing this kind of thing, I usually use TSV instead of CSV and end up using Unix tools anyway. None of the CSV tools keep up in terms of performance.
Re:Cheating at a company group activity using Unix tools by devenvdev on Monday December 13, 2021 @05:39AM (#29538400)
Yes! Agree. This post is more educational than practical (albeit the title). You need to start somewhere and know the basics to understand these details and caveats to feel what quality code means in this context. I chose to use the long, redundant version to show that chaining a gazillion different commands is ok.
Re:Cheating at a company group activity using Unix tools by mdoms on Monday December 13, 2021 @01:10PM (#29542398)
It will also fall flat on its face for CSV files that contain values which contain escaped commas.