Re:Learn to use Awk with hundreds of examples
by asicsp on Wednesday October 25, 2017 @10:42AM
(#15550503)
No need so many commands :)
$ cat duplicates.txt
abc 7 4
food toy ****
abc 7 4
test toy 123
good toy ****
$ awk '!seen[$2]++' duplicates.txt
abc 7 4
food toy ****
$ awk '!seen[$2]++{cnt++} END{print +cnt}' duplicates.txt
2
Re:Learn to use Awk with hundreds of examples
by kazinator on Wednesday October 25, 2017 @02:00PM
(#15552423)
On a large file with many duplicates, seen[x]++ can overflow, unless you're using GNU Awk with bignums (gawk -M).
Re:Learn to use Awk with hundreds of examples
by asicsp on Wednesday October 25, 2017 @10:05PM
(#15555817)
that's a good point
I'll add a note, thanks :)
Re:Learn to use Awk with hundreds of examples
by ninkendo on Wednesday October 25, 2017 @10:29AM
(#15550385)
Dumb side note: I always end up using `awk '{print $1}'` instead of `cut -f1` because cut's field separation defaults are unwieldy in that it doesn't intelligently consider any whitespace to be a field separator, which is what I need 99% of the time.
Re:Learn to use Awk with hundreds of examples
by ptman on Wednesday October 25, 2017 @12:33PM
(#15551583)
sometimes (if there are not mixed spaces and tabs) tr -s ' ' < input | cut -f1 helps
Re:Learn to use Awk with hundreds of examples
by dredmorbius on Wednesday October 25, 2017 @10:55AM
(#15550622)
If you just want to count unique IPs, you can get that in awk alone and avoid sort:
awk '{ip[$1]=1}; END{ print length(ip) }'
Use the hash tables directly, assuming you've got the memory. That should be slightly faster as it avoids the sort.
Re:Learn to use Awk with hundreds of examples
by asicsp on Wednesday October 25, 2017 @11:06AM
(#15550711)
nice one
also, {ip[$1]} is enough...
Re:Learn to use Awk with hundreds of examples
by dredmorbius on Wednesday October 25, 2017 @03:56PM
(#15553548)
Thanks, I was trying to think of the fastest possible assignment.
Re:Learn to use Awk with hundreds of examples
by dietr1ch on Wednesday October 25, 2017 @10:23AM
(#15550327)
There's `sort -u` which should be able to avoid some work.
Me too. For example, number of unique IP requests in a log file in milliseconds ;)
$ awk '{ print $1 } ' caddy_log | sort | uniq | wc -l
But rarely compose my own from scratch. It's mostly copy paste. And store in admin bin for future use.