Re:Command-line tools can be faster than your Hadoop cluster
by azylman on Sunday January 18, 2015 @03:56PM
(#8908846)
What toolset are you using that you can run both locally and on a Hadoop cluster?
[ Parent
| on HN ]
Re:Command-line tools can be faster than your Hadoop cluster
by mdaniel on Sunday January 18, 2015 @05:05PM
(#8909122)
Almost all of them?
The vocabulary of the grandparent comment implies they are using hadoop's streaming mode, and thus one can use a map-reduce streaming abstraction such as MRJob or just plain stdin/stdout; both will work locally and in cluster mode.
Or, if static typing is more agreeable to your development process, running hadoop in "single machine cluster" mode is relatively painless. The same goes for other distributed processing frameworks like Spark.