Modernizing AWK, a 45-year old language, by adding CSV support

Re:Modernizing AWK, a 45-year old language, by adding CSV su... by mattewong on Wednesday May 11, 2022 @04:14PM (#31345149)

FYI I ran on the worldcities data at https://github.com/petewarden/dstkdata (credit to xsv for choosing that dataset) against https://github.com/BurntSushi/xsv and https://github.com/liquidaty/zsv (full disclosure: I am one of the zsv authors). Here's what I got.

fastest to slowest: zsv (0.07), xsv (0.16), goawk (0.42), python (~1.6)

Obviously, does not tell the whole story as this test was limited to "count" and an interpreted language is expected to always be slower compared to a precompiled command, but, it might be relevant to a user deciding what tool to use. Also, might be instructive as to room for improvement in the go code (or possibly the go code could use the c lib)-- I note that even if the goawk command is '{}' the runtime is still about the same.

full results:

---

goawk:

1000001 7000007 real 0m0.435s user 0m0.435s sys 0m0.031s

1000001 7000007 real 0m0.413s user 0m0.419s sys 0m0.024s

1000001 7000007 real 0m0.425s user 0m0.430s sys 0m0.024s

xsv:

1000000 real 0m0.157s user 0m0.141s sys 0m0.013s

1000000 real 0m0.156s user 0m0.141s sys 0m0.012s

1000000 real 0m0.158s user 0m0.142s sys 0m0.013s

zsv:

1000000 real 0m0.066s user 0m0.053s sys 0m0.010s

1000000 real 0m0.077s user 0m0.060s sys 0m0.012s

1000000 real 0m0.069s user 0m0.056s sys 0m0.010s

python:

1000001 7000007 real 0m1.589s user 0m1.553s sys 0m0.026s

1000001 7000007 real 0m1.583s user 0m1.550s sys 0m0.025s

1000001 7000007 real 0m2.122s user 0m1.675s sys 0m0.037s

---

The script for this was:

---

echo 'goawk:'

(time goawk -i csv '{ w+=NF } END { print NR, w }' < worldcitiespop_mil.csv) 2>&1 | xargs

echo 'xsv:'

(time xsv count < worldcitiespop_mil.csv) 2>&1 | xargs

echo 'zsv:'

(time zsv count < worldcitiespop_mil.csv) 2>&1 | xargs

echo 'python:'

(time python3 count.py < worldcitiespop_mil.csv) 2>&1 | xargs

[ Parent | on HN ]

Re:Modernizing AWK, a 45-year old language, by adding CSV su... by benhoyt on Wednesday May 11, 2022 @05:35PM (#31346134)

Thanks for that! zsv looks amazing. Yeah, it's definitely going to whip GoAWK, what with SIMD parsing and careful memory handling. I've done a couple of basic things for GoAWK's CSV performance, but haven't profiled or looked at allocation bottlenecks (absolute performance definitely wasn't my first focus).

Yeah, sorry about huge.csv -- I found it online somewhere originally by searching for something like "large csv example", but can't for the life of me find it now. It's a monster 1.5GB file with 286 columns including quoted fields (whereas worldcitiespop only has a few columns and it doesn't look like it has quoted fields). I can upload to a file transfer service and send a link to you if you want ... though I should really update my benchmarks to use an easily-downloadable file instead.

[ Parent | on HN ]

Re:Modernizing AWK, a 45-year old language, by adding CSV su... by mattewong on Thursday May 12, 2022 @10:59AM (#31355203)

Not a problem at all. Nice work with goawk!

[ Parent | on HN ]

Re:Modernizing AWK, a 45-year old language, by adding CSV su... by benhoyt on Friday May 13, 2022 @07:41AM (#31366189)

I've now updated the benchmarks to avoid huge.csv: the write benchmarks now write a big 1GB, 20-column CSV and the read benchmarks use that same CSV: https://github.com/benhoyt/goawk/commit/07eb4505a7f64cffceeb...

[ Parent | on HN ]