News for hackers. Stuff that matters.
Re:Benchmarking shell pipelines and the Unix “tools” philosophy by dredmorbius on Wednesday January 08, 2020 @03:48AM (#21989626)
Even working in memory, there are different efficiencies for different methods.

Awk includes an asort() function which can sort an array, such that it would be possible to create a similar process entirely within awk to the sort | uniq -c pipeline:

    #!/usr/bin/gawk -f
    { x[NR] = $1 }
    END {
        rc = asort(x)
        j=0
        for(i in x) {
            if( x[i] "" == x[i-1] "" ) freq[j]++
     else {
         j++
         elem[j] = x[i]
         freq[j] = 1
     }
        }
        for(j in elem) {
            printf( "%6i %s\n", freq[j], elem[j])
        }
    }
As compares with a hash-based counter:

    #!/usr/bin/gawk -f
    { x[$1]++ }
    END { for(i in x ) printf( "%6i %s\n", x[i], i ) }
On a 2,000 value test dataset with 10 unique values:

sort | uniq -c takes 0.019s (8 runs averaged)

awk hash takes 0.023s (8 runs averaged)

awk-implemented sort + unique takes 0.33s (8 runs averaged)

In this case, sort | uniq is the fastest option. But the all-in-memory sort + separate tabulation of unique values in awk is notably slower (running in 143% of the time) than the also all-in-memory hash accumulator.

As I bump up the dataset size (20,000 records) that discrepency increases, roughly 0.052s sort|uniq, 0.065s hash, and 0.217s ask sort-unique.

TL;DR: test your assumptions, especially regarding performance.

Note: Data were generated with a simple bash loop:

    for i in {1..2000}; do echo $((RANDOM%10)); done  > data
Re:Benchmarking shell pipelines and the Unix “tools” philosophy by justinsaccount on Wednesday January 08, 2020 @10:41AM (#21992337)
With super small dataset sizes like that it'll fit in the L2 cache and can behave differently. I did test this though, and for 20,000 items generated with your loop:

sort | uniq -c takes .017s (fastest out of a few runs)

the awk command I used above takes .013s

A trivial implementation I have in go takes .08s

Additionally, using this 'protos' file which is 1,000,000 lines of tcp,udp,icmp:

  $ time (sort protos|uniq -c)
     5915 icmp
   332003 tcp
   662082 udp

  real 0m0.232s
  user 0m0.739s
  sys 0m0.100s

  $ # fixed to count 'lines' and not the first column, which makes it faster.
  $ time awk '{lines[$0]++} END {for (l in lines) printf("%s %d\n", l, lines[l])}'  < protos
  icmp 5915
  udp 662082
  tcp 332003

  real 0m0.194s
  user 0m0.190s
  sys 0m0.004s

  $ time ./c < protos
  662082 udp
  332003 tcp
  5915 icmp

  real 0m0.088s
  user 0m0.084s
  sys 0m0.005s
so yes, I do test my assumptions.
Re:Benchmarking shell pipelines and the Unix “tools” philosophy by dredmorbius on Wednesday January 08, 2020 @03:03PM (#21995172)
I was suggesting @crystaldev may not have.

And, more significantly, and as you've confirmed, not all in-memory processing is equivalent. There are faster and slower all-in-memory algorithms and implementations

Hrm... Maybe a bogounique implementation might be appropriate here....

(By analogy to bogosort: https://en.wikipedia.org/wiki/Bogosort)