Re:MapReduce for C: Run Native Code in Hadoop
by choppaface on Wednesday February 18, 2015 @08:01PM
(#9071893)
Hmmm, this is definitely not some sort of vanilla C interface for Hadoop. MR4C is rather a ~40ksloc C++/Java library capable of performing somewhat high-level operations on datasets. It's closer to a MapReduce app or a tool like Pig than a simple C/C++ wrapper. In particular, MR4C has its own data model:
... and a 3600 line C++ package for extracting crops from satellite images based upon geospatial coordinates (that is likely more relevant to Skybox than Hadoop users)
* https://github.com/google/mr4c/tree/master/tutorial/UserGuid...
... and nontrivial logging, traceback debugging, tempfile handling.. stuff for writing a library in MR4C
* https://github.com/google/mr4c/blob/master/native/src/cpp/im...
* https://github.com/google/mr4c/blob/master/native/src/cpp/im...
* https://github.com/google/mr4c/blob/master/native/src/cpp/im...
... and a 3600 line C++ package for extracting crops from satellite images based upon geospatial coordinates (that is likely more relevant to Skybox than Hadoop users)
* https://github.com/google/mr4c/tree/master/geospatial
... and singletons all over the place! (Kenton Varda would not be happy)
* https://github.com/google/mr4c/blob/master/native/src/cpp/ap...
Props to the Skybox devs for getting this out. All of this code almost certainly got thrown away as part of Skybox's transition onto the Google stack.
Have not yet found any benchmarks or illustrations of I/O savings.