Re:How to Analyze Billions of Records per Second on a Single...
by mmt on Tuesday July 10, 2018 @04:57PM
(#17502084)
It's probably easy enough to simulate the "bash" implementation done in a distributed way. Doing that python library full parsing locally, 10,000 lines (or however much memory one can configure up to) at a time might also be easy enough.
Come to think of it, just benchmarking a single-threaded run of enough lines of each against each other should give you an idea, especially with even basic profiling from time(1) to estimate how much was CPU and how much was I/O.
That said, I don't think that such a discrepancy, if it exists, takes away from what I took to be the author's overall point, which is that old, simple tools, are underused, despite being much faster and good enough, i.e. worse is better [1].
[1] This an example of sacrificing correctness for simplicity in implementation, something detailed in the "rise of worse is better" essay, not merely an excuse for sloppiness in engineering.
Come to think of it, just benchmarking a single-threaded run of enough lines of each against each other should give you an idea, especially with even basic profiling from time(1) to estimate how much was CPU and how much was I/O.
That said, I don't think that such a discrepancy, if it exists, takes away from what I took to be the author's overall point, which is that old, simple tools, are underused, despite being much faster and good enough, i.e. worse is better [1].
[1] This an example of sacrificing correctness for simplicity in implementation, something detailed in the "rise of worse is better" essay, not merely an excuse for sloppiness in engineering.