Command-line tools can be faster than your Hadoop cluster

Re:Command-line tools can be faster than your Hadoop cluster by Symbiote on Sunday January 18, 2015 @07:08PM (#8909565)

'pv' shows a progress bar, something like '&& touch $x.success' can help restart recovery.

I'd probably pick the shell approach for something I expect to be a one-off, but reconsider each time the task is repeated.

I printed http://xkcd.com/1205/ and stuck it to the wall. It's a useful reference when someone seems to be under or overengineering something.

[ Parent | on HN ]

Re:Command-line tools can be faster than your Hadoop cluster by lloeki on Monday January 19, 2015 @03:46AM (#8911013)

While it drives some point home, the chart eludes the question of robustness (a written script will run twice the same way, whereas human error, especially on routine tasks, may hit one hard) and documentation (writing even a lightly commented script to do yearly maintenance is guaranteed to help your future self remember things one year from now).

[ Parent | on HN ]

Re:Command-line tools can be faster than your Hadoop cluster by slagfart on Sunday January 18, 2015 @09:54PM (#8910045)

That chart assumes 24 hours days. The reality of (my?) productivity is that I have perhaps six productive hours in a day. If I can save eight productive hours per month, that's sixteen days a year, not four.

[ Parent | on HN ]