News for hackers. Stuff that matters.
Re:Learn to use Awk with hundreds of examples by 0xFFFE on Wednesday October 25, 2017 @10:02AM (#15550179)
I use sed & awk all the time. It is an invaluable tool while debugging issues, extracting fields from log files etc. I am not dissing python or perl, I use python extensively as well. But while you are in the middle of an incident, hard to beat a quick oneliner, like this random example.

awk -F":" 'BEGIN{total=0}{if($3>240)total+=$3}END{print total}' /etc/passwd

Re:Learn to use Awk with hundreds of examples by ajross on Wednesday October 25, 2017 @10:49AM (#15550582)
Interestingly, perl was designed to fill exactly that role, with a syntax that was at the time considered cleaner and more intuitive than awk's (and of course with a regex backend that was much more powerful). In this case, something like:

    perl -e 'while(<>) { $tot += (split)[2]; }; print "$tot\n"'
It's funny to see that in perl's decline, the stuff it did well is being forgotten and resurrected via tools it at one point had mostly replaced.
Re:Learn to use Awk with hundreds of examples by kbenson on Wednesday October 25, 2017 @12:36PM (#15551606)
Which is a shame, because I doubt I'll be loading modules from CPAN for awk any time soon, and I routinely use modules from CPAN or that I myself have written in my perl one liners. In fact, I have a specialized project-perl.sh in my project's PATH that loads a few modules (Try::Tiny, Path::Tiny, Data::Dumper, etc), sets a local::lib, and loads the main project module, allowing access to most the functions I've written for the current project, and passes all the extra params through to the underlying perl command.

If my codebase wasn't already in perl I wouldn't necessarily be able to benefit from a large library of my own functions relevant to the problem, but at least I would still have CPAN to fall back on. I'm not sure awk will ever be able to compete with that.

Re:Learn to use Awk with hundreds of examples by majewsky on Wednesday October 25, 2017 @11:01AM (#15550668)
Let me golf this. :)

  perl -F: -anE '$tot+=$F[2]}{say$tot'
Re:Learn to use Awk with hundreds of examples by jcims on Wednesday October 25, 2017 @12:19PM (#15551457)
This made me miss perl (but ya forgot $F[2]>240)
Re:Learn to use Awk with hundreds of examples by majewsky on Thursday October 26, 2017 @05:44AM (#15557529)
Ah, I was going by the snippet in the parent, not the original one in the grandparent. That here should do it:

  perl -F: -anE '$_=$F[2];$t+=$_if$_>240}{say$t'
I wanted to be clever by doing `$t+=$_*($_>240)`, but that's actually one byte larger. :(
Re:Learn to use Awk with hundreds of examples by asicsp on Wednesday October 25, 2017 @11:00AM (#15550659)
Perl does have command-line options to do that :)

    perl -F: -lane '$total+=$F[2] if $F[2]>240; END{print $total}' /etc/passwd
Re:Learn to use Awk with hundreds of examples by ajross on Wednesday October 25, 2017 @12:01PM (#15551237)
Of course it does, the implicit looping and BEGIN/END blocks (I'll concede I had actually forgotten about the implicit splitting) were in fact deliberately designed to emulate awk. Nonetheless I don't think that really changes the point much. The core language syntax is simple enough that perl one-liners can be written without that and still get the point across.
Re:Learn to use Awk with hundreds of examples by indescions_2017 on Wednesday October 25, 2017 @10:13AM (#15550264)
>>> I use sed & awk all the time.

Me too. For example, number of unique IP requests in a log file in milliseconds ;)

$ awk '{ print $1 } ' caddy_log | sort | uniq | wc -l

But rarely compose my own from scratch. It's mostly copy paste. And store in admin bin for future use.

Re:Learn to use Awk with hundreds of examples by asicsp on Wednesday October 25, 2017 @10:42AM (#15550503)
No need so many commands :)

    $ cat duplicates.txt
    abc  7   4
    food toy ****
    abc  7   4
    test toy 123
    good toy ****

    $ awk '!seen[$2]++' duplicates.txt
    abc  7   4
    food toy ****

    $ awk '!seen[$2]++{cnt++} END{print +cnt}' duplicates.txt
    2
Re:Learn to use Awk with hundreds of examples by kazinator on Wednesday October 25, 2017 @02:00PM (#15552423)
On a large file with many duplicates, seen[x]++ can overflow, unless you're using GNU Awk with bignums (gawk -M).
Re:Learn to use Awk with hundreds of examples by asicsp on Wednesday October 25, 2017 @10:05PM (#15555817)
that's a good point

I'll add a note, thanks :)

Re:Learn to use Awk with hundreds of examples by ninkendo on Wednesday October 25, 2017 @10:29AM (#15550385)
Dumb side note: I always end up using `awk '{print $1}'` instead of `cut -f1` because cut's field separation defaults are unwieldy in that it doesn't intelligently consider any whitespace to be a field separator, which is what I need 99% of the time.
Re:Learn to use Awk with hundreds of examples by ptman on Wednesday October 25, 2017 @12:33PM (#15551583)
sometimes (if there are not mixed spaces and tabs) tr -s ' ' < input | cut -f1 helps
Re:Learn to use Awk with hundreds of examples by dredmorbius on Wednesday October 25, 2017 @10:55AM (#15550622)
If you just want to count unique IPs, you can get that in awk alone and avoid sort:

    awk '{ip[$1]=1}; END{ print length(ip) }'
Use the hash tables directly, assuming you've got the memory. That should be slightly faster as it avoids the sort.
Re:Learn to use Awk with hundreds of examples by asicsp on Wednesday October 25, 2017 @11:06AM (#15550711)
nice one

also, {ip[$1]} is enough...

Re:Learn to use Awk with hundreds of examples by dredmorbius on Wednesday October 25, 2017 @03:56PM (#15553548)
Thanks, I was trying to think of the fastest possible assignment.
Re:Learn to use Awk with hundreds of examples by dietr1ch on Wednesday October 25, 2017 @10:23AM (#15550327)
There's `sort -u` which should be able to avoid some work.
Re:Learn to use Awk with hundreds of examples by c4n4rd on Wednesday October 25, 2017 @10:58AM (#15550649)
No need to spawn another process(uniq), you can use sort -u

;-)

Re:Learn to use Awk with hundreds of examples by kbenson on Wednesday October 25, 2017 @12:41PM (#15551657)
For me, the benefit of a trailing uniq instead of a sort -u is that I'm invariably building towards a "sort | uniq -c | sort -n" anyways.
Re:Learn to use Awk with hundreds of examples by vram22 on Wednesday October 25, 2017 @10:53AM (#15550617)
The fragment

BEGIN{total=0}

can be skipped (at least in the awk's I've used, not sure if some more recent / strict awk differs), because awk initializes the variable total to 0.

I remember this because I do this all the time to quickly get the (non-recursive) sum of the sizes of the files in a directory - it's pretty much muscle memory from a while now:

ls -l | awk '{ s += $5 } END { print s/1024 " KB" }'

For recursive size, one can use ls -lR or the du command with various options according to need.

Re:Learn to use Awk with hundreds of examples by kazinator on Wednesday October 25, 2017 @02:02PM (#15552450)
But that only works because in the final print, you have done arithmetic with s.

So in other words, we can take out the BEGIN block, but then we must remember to change print total to print total + 0.

Also, using uninitialized variables is basically a code golfing stupidity that will bite you in any halfway complicated program.

GNU Awk has a useful --lint argument which spots uses of uninitialized variables. If you make it habit to write code that way, if you then use --lint for finding a bug, you have to deal with false positives.

Re:Learn to use Awk with hundreds of examples by vram22 on Wednesday October 25, 2017 @03:18PM (#15553222)
>Also, using uninitialized variables is basically a code golfing stupidity that will bite you in any halfway complicated program.

Nonsense. Not if you know what you are doing, and used it in a known way, which is what I did. The code I wrote works. I tested it on Linux before posting it. Also, such a usage (skipping the initializer) is mentioned (IIRC) in the classic Kernighan & Pike book "The Unix Programming Environment" (still a great resource, though not updated for modern Unix/Linux features), which is where I learned it from, years ago (and hence why I qualified my statement by saying it may not work in more strict or modern awk versions). Fine to talk about other variations but it does not mean that my variation is wrong.

Don't try to read my mind. My intention was not code golfing. Was just sharing some fun info. It's not a big deal to keep the initializer either, I'm quite aware of that.

Re:Learn to use Awk with hundreds of examples by kazinator on Wednesday October 25, 2017 @04:21PM (#15553808)
You do not know 100% what you're doing. Evidence being, in the grandparent comment you wrote "[...] because awk initializes the variable total to 0" which isn't how awk works at all.

Your intention can be understood as the promotion of code golfing, as evidenced by these words:

The fragment BEGIN{total=0} can be skipped

by which you're clearly encouraging that other coder to make their code shorter by removing an initialization that works fine.

Re:Learn to use Awk with hundreds of examples by vram22 on Wednesday October 25, 2017 @06:07PM (#15554734)
>Your intention can be understood as the promotion of code golfing, as evidenced by these words: The fragment BEGIN{total=0} can be skipped by which you're clearly encouraging that other coder to make their code shorter by removing an initialization that works fine.

Your statement (above) can be "understood" as not understanding my prior statement(s), including the one in which I said "Don't try to read my mind" (w.r.t. intention, because you cannot - it is mine (mind), not yours). If you cannot grok that after a second explanation, I have nothing further to say. Good day.

Re:Learn to use Awk with hundreds of examples by kazinator on Wednesday October 25, 2017 @06:36PM (#15554898)
Although your true intent may be anyone's guess, true enough, the prima faciae intent of forum posting persona is a fairly straightforward function of the content via which it portrays itself.
Re:Learn to use Awk with hundreds of examples by vram22 on Wednesday October 25, 2017 @05:35PM (#15554481)
>code golfing stupidity

Smile when you say that, stranger!

http://www.thisdayinquotes.com/2011/05/when-you-call-me-that...

Re:Learn to use Awk with hundreds of examples by feelin_googley on Wednesday October 25, 2017 @10:44PM (#15555960)

  #!/bin/sh
  x=0;while true;do read a;
  test ${#a} -gt 0||exec echo $x
  a=${a#*:\*:};n=${a%%:*};
  test $n -le 240||x=$((x+n));
  done < /etc/passwd
Re:Learn to use Awk with hundreds of examples by feelin_googley on Thursday October 26, 2017 @11:10AM (#15559516)

  #!/bin/sh
  IFS=:;x=0;while true;do read a b c d;
  test ${#a} -gt 0||exec echo $x
  test $c -lt 240||x=$((x+c));
  done < /etc/passwd
Re:Learn to use Awk with hundreds of examples by asicsp on Wednesday October 25, 2017 @10:49AM (#15550583)
you could simplify by taking advantage of default value which is `0` in numeric context

    awk -F: '$3>240{total+=$3} END{print +total}' /etc/passwd