News for hackers. Stuff that matters.
Re:The 2016 Top Programming Languages by doug1001 on Thursday July 28, 2016 @05:56AM (#12179328)
nice one. small, sharp tools in the unix toolbox. given that this is the type of processing one might routinely over a file with tens of millions of lines, what's your guess on the difference in performance between the python snippet above your your awk one-liner? (my guess is about 100x in favor of awk)
Re:The 2016 Top Programming Languages by claystu on Thursday July 28, 2016 @09:09AM (#12179929)
I honestly have no idea. Given that there are different versions of awk and different versions of python, I'm not even sure there is an answer.

Given awk's age (1977-when computers were much slower and memory much more expensive) and pedigree (Aho, Weinberger, and Kernighan), I wouldn't bet against it for a task like you describe, but that's just a feeling. Again, I don't have any numbers to support that.

Re:The 2016 Top Programming Languages by iheartmemcache on Thursday July 28, 2016 @09:12AM (#12179941)
My gut-ballpark was going to be around an order of magnitude, not two. Here's are two naive comparisons (granted, from the late aughts and not a direct comparison but a more general one) that show ~7-8x[0,1]. The overhead of PyStringObject is not trivial[2] (though the implementation details likely have changed between Py2.x and Python 3).

For things like building accumulators a set of data/log parsing rather than data munging (hits per hour or enumerative tasks), I'd imagine (g|n)?awk might hit your 100x since you'd just grab the fd and traverse being IO bound. I'm not sure how awk does it, but if it's just saving an accumulator value (or 10) in a register. Assuming x64-64 treats, say, an r3 fetch analogously to a fetch to ecx (err..rcx now I guess), rather than having to keep a full object in L1 cache, awk has a huge advantage.

---

N.b., if you're benching tasks like this, don't use 'time' and STDOUT and think you're getting real performance numbers. Your bottleneck (terminals can only render $x lines a minute, so the kernel call to write(STDOUT, ....) will be where you choke, not at the language. Also disk fragmentation would be another issue. Put the both the test file and the output file on a RAMdisk.) Cache flush with Something like sync; echo 3 > /proc/sys/vm/drop_caches (on Linux, I forget the BSD way of doing it) then `time benchmark.py /mnt/ramdisk1/file > /mnt/ramdisk2' over multiple runs, under various loads, with different data sets, etc

Another interesting thing to note is that comp arch is so advanced (I was reading a paper on formal verification of ISAs, and apparently even 12 dollar ARMs now have out-of-order instruction execution type stuff) that between the kernel scheduler and CPU optimizations, Python will likely benefit much more from disk-seek latency (effectively allowing PyStringObject allocation to occur while you're waiting for /dev/sd$n to return).

I'm certainly not an authority on rigorous benchmarks though - someone like Brendan Gregg please jump in!

[0] https://diamondinheritance.blogspot.com/2008/04/awk-vs-pytho...

[1] https://brenocon.com/blog/2009/09/dont-mawk-awk-the-fastest-...

[2] http://www.laurentluce.com/posts/python-string-objects-imple...

Re:The 2016 Top Programming Languages by doug1001 on Saturday July 30, 2016 @12:26AM (#12191641)
nice one--i learned a few things, in fact (& caused me to realize once again, just how sloppy we are with benchmarking on my team)