News for hackers. Stuff that matters.
Re:Command-line tools can be faster than your Hadoop cluster by dice on Sunday January 18, 2015 @04:28PM (#8908968)
Keep reading, he removes the cat and grep in the final solution.
Re:Command-line tools can be faster than your Hadoop cluster by omaranto on Sunday January 18, 2015 @08:04PM (#8909750)
Yes, but he still keeps the awkward Awk code with the substr and such. I haven't benchmarked, maybe that's faster than the pretty regex matches.
Re:Command-line tools can be faster than your Hadoop cluster by lloeki on Monday January 19, 2015 @03:55AM (#8911040)
I believe this is to be a bit more educative about how to build a pipeline. Also, iteratively building such solutions quickly often leads to such "inefficiencies" but makes things easier to reason with. Besides, the awk step may have been factored out in the end so it wouldn't make sense to optimise early. Also, by the time the author reaches the end, he gets IO-bound so there's not much need to optimise further (in the context of the exercise).