News for hackers. Stuff that matters.
Re:Learn to use Awk with hundreds of examples by indescions_2017 on Wednesday October 25, 2017 @10:13AM (#15550264)
>>> I use sed & awk all the time.

Me too. For example, number of unique IP requests in a log file in milliseconds ;)

$ awk '{ print $1 } ' caddy_log | sort | uniq | wc -l

But rarely compose my own from scratch. It's mostly copy paste. And store in admin bin for future use.

Re:Learn to use Awk with hundreds of examples by asicsp on Wednesday October 25, 2017 @10:42AM (#15550503)
No need so many commands :)

    $ cat duplicates.txt
    abc  7   4
    food toy ****
    abc  7   4
    test toy 123
    good toy ****

    $ awk '!seen[$2]++' duplicates.txt
    abc  7   4
    food toy ****

    $ awk '!seen[$2]++{cnt++} END{print +cnt}' duplicates.txt
    2
Re:Learn to use Awk with hundreds of examples by kazinator on Wednesday October 25, 2017 @02:00PM (#15552423)
On a large file with many duplicates, seen[x]++ can overflow, unless you're using GNU Awk with bignums (gawk -M).
Re:Learn to use Awk with hundreds of examples by asicsp on Wednesday October 25, 2017 @10:05PM (#15555817)
that's a good point

I'll add a note, thanks :)

Re:Learn to use Awk with hundreds of examples by ninkendo on Wednesday October 25, 2017 @10:29AM (#15550385)
Dumb side note: I always end up using `awk '{print $1}'` instead of `cut -f1` because cut's field separation defaults are unwieldy in that it doesn't intelligently consider any whitespace to be a field separator, which is what I need 99% of the time.
Re:Learn to use Awk with hundreds of examples by ptman on Wednesday October 25, 2017 @12:33PM (#15551583)
sometimes (if there are not mixed spaces and tabs) tr -s ' ' < input | cut -f1 helps
Re:Learn to use Awk with hundreds of examples by dredmorbius on Wednesday October 25, 2017 @10:55AM (#15550622)
If you just want to count unique IPs, you can get that in awk alone and avoid sort:

    awk '{ip[$1]=1}; END{ print length(ip) }'
Use the hash tables directly, assuming you've got the memory. That should be slightly faster as it avoids the sort.
Re:Learn to use Awk with hundreds of examples by asicsp on Wednesday October 25, 2017 @11:06AM (#15550711)
nice one

also, {ip[$1]} is enough...

Re:Learn to use Awk with hundreds of examples by dredmorbius on Wednesday October 25, 2017 @03:56PM (#15553548)
Thanks, I was trying to think of the fastest possible assignment.
Re:Learn to use Awk with hundreds of examples by dietr1ch on Wednesday October 25, 2017 @10:23AM (#15550327)
There's `sort -u` which should be able to avoid some work.
Re:Learn to use Awk with hundreds of examples by c4n4rd on Wednesday October 25, 2017 @10:58AM (#15550649)
No need to spawn another process(uniq), you can use sort -u

;-)

Re:Learn to use Awk with hundreds of examples by kbenson on Wednesday October 25, 2017 @12:41PM (#15551657)
For me, the benefit of a trailing uniq instead of a sort -u is that I'm invariably building towards a "sort | uniq -c | sort -n" anyways.