Scripting with Go: A Modest Proposal

Re:Scripting with Go: A Modest Proposal by sgarland on Sunday August 20, 2023 @02:00PM (#37201485)

Collation aside (which is absolutely a huge boost in speed that I neglected to think about), I assumed that the rest of the difference was coming from the fact that the initial `cut` meant the rest of the pipeline had far less to deal with, whereas `awk` is processing every line. Benchmarking (and testing in `perf`) showed this to not be the case. I'd need to compile `awk` with debug symbols, I think, to know exactly where the slowdown is, but I'm going to assume it's mostly due to `sort` being extremely optimized for doing one thing, and doing it well.

I did find one other interesting difference between BSD and GNU tools - BSD sort defaults to 90% for its buffer, GNU sort defaults to 1024 KiB.

Combining all of these (and using GNU uniq - it was also faster), I was able to get down to 463 msec on the M1 Air:

  $ hyperfine "export LC_ALL=C; gcut -d' ' -f1 fake_log.txt | gsort -S5% | guniq -c | gsort -rn -S5% | head"
  Benchmark 1: export LC_ALL=C; gcut -d' ' -f1 fake_log.txt | gsort -S5% | guniq -c | gsort -rn -S5% | head
  Time (mean ± σ):     463.4 ms ±   3.3 ms    [User: 965.5 ms, System: 93.3 ms]
  Range (min … max):   459.9 ms … 469.8 ms    10 runs

TIL, thank you.

[ Parent | on HN ]

Re:Scripting with Go: A Modest Proposal by xvector on Sunday August 20, 2023 @05:12PM (#37202974)

Could you elaborate on how you arrived at 5% for your buffer? Does specifying a buffer size really cause that much of a speed up?

[ Parent | on HN ]

Re:Scripting with Go: A Modest Proposal by sgarland on Sunday August 20, 2023 @07:09PM (#37203811)

Parent poster used the 5% value, so I did as well. But yes, to answer your question. Shown here on a Debian 11 system that's fairly old:

  $ hyperfine "export LC_ALL=C; cut -d' ' -f1 fake_log.txt | sort -S5% | uniq -c | sort -rn -S5% | head"
  Benchmark 1: export LC_ALL=C; cut -d' ' -f1 fake_log.txt | sort -S5% | uniq -c | sort -rn -S5% | head
  Time (mean ± σ):      1.504 s ±  0.318 s    [User: 2.833 s, System: 0.474 s]
  Range (min … max):    0.942 s …  1.937 s    10 runs
 
  $ hyperfine "export LC_ALL=C; cut -d' ' -f1 fake_log.txt | sort | uniq -c | sort -rn | head"
  Benchmark 1: export LC_ALL=C; cut -d' ' -f1 fake_log.txt | sort | uniq -c | sort -rn | head
  Time (mean ± σ):      3.847 s ±  0.093 s    [User: 4.165 s, System: 0.613 s]
  Range (min … max):    3.591 s …  3.919 s    10 runs

Setting the buffer value to ~half the file size (100 MB) resulted in a mean time of 2.291 seconds. Setting it to the size of the file resulted in a mean time of 1.549 seconds, which is close enough to the 5% to call it equal - not like this server isn't busy with other stuff, so it's hardly a good place for perfect benchmarking.

Syscalls aren't free, nor are disk reads, so if you have the RAM to support slurping the entire file at once, it's sometimes faster.

[ Parent | on HN ]