Re:How to Analyze Billions of Records per Second on a Single...
by mmt on Tuesday July 10, 2018 @04:57PM
(#17502084)
It's probably easy enough to simulate the "bash" implementation done in a distributed way. Doing that python library full parsing locally, 10,000 lines (or however much memory one can configure up to) at a time might also be easy enough.
Come to think of it, just benchmarking a single-threaded run of enough lines of each against each other should give you an idea, especially with even basic profiling from time(1) to estimate how much was CPU and how much was I/O.
That said, I don't think that such a discrepancy, if it exists, takes away from what I took to be the author's overall point, which is that old, simple tools, are underused, despite being much faster and good enough, i.e. worse is better [1].
[1] This an example of sacrificing correctness for simplicity in implementation, something detailed in the "rise of worse is better" essay, not merely an excuse for sloppiness in engineering.
In that article, the author runs a regular expression that looks lines that start with "Result" in the pgn files. Then they do a tiny bit of processing with awk.
In the original article[0] the author completely parses the pgn files. Then they extract the result from the parsed file.
I wouldn't be surprised if the difference came completely from that.
I would love to see someone run benchmark the exact same program and see how much of a difference there really is between Hadoop and Bash.
[0] https://web.archive.org/web/20140119221101/http://tomhayden3...