News for hackers. Stuff that matters.
Re:Seven Unix Commands Every Data Scientist Should Know by 6keZbCECT2uB on Thursday March 07, 2019 @12:08AM (#19325662)
tldr: grep, cat, find, head / tail, wc, awk, shuf with bonuses of xargs, and man.

I've never needed shuf, and awk is a bit out of place in the list, but head and tail have saved me from many a large file. The interesting data is usually in head, tail, or grep anyway.

Re:Seven Unix Commands Every Data Scientist Should Know by vajrabum on Thursday March 07, 2019 @12:55AM (#19325859)
awk '{print $2,$4;}' is useful and easy to remember. $NF refers to the last field. FS is the variable to override the default field separator of white space. Here's an example of it's use. OFS is the output field separator.

awk 'BEGIN{FS=":"; OFS=":"} {print $1,$NF}' /etc/password

Works on CSV files too with the right field separator.

Re:Seven Unix Commands Every Data Scientist Should Know by masklinn on Thursday March 07, 2019 @02:16AM (#19326148)
> awk '{print $2,$4;}' is useful and easy to remember. $NF refers to the last field. FS is the variable to override the default field separator of white space. Here's an example of it's use. OFS is the output field separator.

Yeah also cut is just inconvenient, awk works much better when working with the usual tabulated data.

> Works on CSV files too with the right field separator.

You're probably better off using xsv though.

Re:Seven Unix Commands Every Data Scientist Should Know by Noumenon72 on Thursday March 07, 2019 @10:25PM (#19334853)
I have used awk several times in different ways. I still don't find it easy to remember a command with as many punctuation characters as letters, like your first one.

And I'm not even sure what the second one is trying to do. Print the user name, followed by the number of fields separated by colons? What good is that? Perhaps you've forgotten that not everyone can read awk syntax right off the page.

Re:Seven Unix Commands Every Data Scientist Should Know by mroche on Thursday March 07, 2019 @12:20AM (#19325704)
Def the first five there. wc as more of a tie-in with the former. Never used shuf and still need to find the time to learn awk.
Re:Seven Unix Commands Every Data Scientist Should Know by scbrg on Thursday March 07, 2019 @01:25AM (#19325966)
When it comes to learning awk, my suggestion is to read the man file of one of the simpler implementations; original-awk[0] or plan9's[1] awk. They lack the GNU additions (many of which admittedly are useful), but they decrease the learning curve a lot. I pushed off learning awk for years since I got overwhelmed every time I typed 'man awk', but after stumbling on the man page from plan9 I learned it in no time. Sure, I don't know all the GNU extensions in gawk, but now I can look them up when I need them.

[0] https://manpages.debian.org/stretch/original-awk/awk.1.en.ht... [1] https://www.unix.com/man-page/plan9/1/AWK/

Re:Seven Unix Commands Every Data Scientist Should Know by mroche on Thursday March 07, 2019 @08:16AM (#19327733)
Thanks for this! I appreciate the pointers.
Re:Seven Unix Commands Every Data Scientist Should Know by emgee_1 on Thursday March 07, 2019 @03:20AM (#19326412)
shut I do not know; would add extra random field and then sort