Every story and comment from the Hacker News API, in a layout borrowed from 2001.
A comment on Choosing a Language for Bioinformatics.
$ awk \ 'BEGIN{for (i=0; i<2000000; i++){print "abcdef\tghijk\tlmnop\tqrstuv\twxyz1234\tABCDEF\tHIJK\tLMNOP\tQRSTUV\tWXYZ123"}}' \ > big.tsv $ cat big.tsv |sort|uniq -c 2000000 abcdef ghijk lmnop qrstuv wxyz1234 ABCDEF HIJK LMNOP QRSTUV WXYZ123
The idea is that if 2 lines are identical, the number of "count++" emitted will be exactly the same.
I took your gawk code, but I am starting the for loop at 2 instead of 1, and I removed the -F'\t' option.
$ time cat big.tsv |sort | uniq -c \ | gawk '{for (i=2; i <= NF; i++) {if (index(tolower(substr($i, 1, 3)), "bc") != 0) {count += $1}}}END{print count}' 4000000 real 0m0.724s user 0m0.605s sys 0m0.322s
There are no comments on this comment yet.
The idea is that if 2 lines are identical, the number of "count++" emitted will be exactly the same.
I took your gawk code, but I am starting the for loop at 2 instead of 1, and I removed the -F'\t' option.
Edit: added backslashes to split lines.