Re:Benchmarking shell pipelines and the Unix “tools” philosophy
by justinsaccount on Wednesday January 08, 2020 @10:41AM
(#21992337)
With super small dataset sizes like that it'll fit in the L2 cache and can behave differently. I did test this though, and for 20,000 items generated with your loop:
sort | uniq -c takes .017s (fastest out of a few runs)
the awk command I used above takes .013s
A trivial implementation I have in go takes .08s
Additionally, using this 'protos' file which is 1,000,000 lines of tcp,udp,icmp:
$ time (sort protos|uniq -c)
5915 icmp
332003 tcp
662082 udp
real 0m0.232s
user 0m0.739s
sys 0m0.100s
$ # fixed to count 'lines' and not the first column, which makes it faster.
$ time awk '{lines[$0]++} END {for (l in lines) printf("%s %d\n", l, lines[l])}' < protos
icmp 5915
udp 662082
tcp 332003
real 0m0.194s
user 0m0.190s
sys 0m0.004s
$ time ./c < protos
662082 udp
332003 tcp
5915 icmp
real 0m0.088s
user 0m0.084s
sys 0m0.005s
so yes, I do test my assumptions.
Re:Benchmarking shell pipelines and the Unix “tools” philosophy
by dredmorbius on Wednesday January 08, 2020 @03:03PM
(#21995172)
I was suggesting @crystaldev may not have.
And, more significantly, and as you've confirmed, not all in-memory processing is equivalent. There are faster and slower all-in-memory algorithms and implementations
Hrm... Maybe a bogounique implementation might be appropriate here....
(By analogy to bogosort: https://en.wikipedia.org/wiki/Bogosort)
Awk includes an asort() function which can sort an array, such that it would be possible to create a similar process entirely within awk to the sort | uniq -c pipeline:
As compares with a hash-based counter: On a 2,000 value test dataset with 10 unique values:sort | uniq -c takes 0.019s (8 runs averaged)
awk hash takes 0.023s (8 runs averaged)
awk-implemented sort + unique takes 0.33s (8 runs averaged)
In this case, sort | uniq is the fastest option. But the all-in-memory sort + separate tabulation of unique values in awk is notably slower (running in 143% of the time) than the also all-in-memory hash accumulator.
As I bump up the dataset size (20,000 records) that discrepency increases, roughly 0.052s sort|uniq, 0.065s hash, and 0.217s ask sort-unique.
TL;DR: test your assumptions, especially regarding performance.
Note: Data were generated with a simple bash loop: