News for hackers. Stuff that matters.
Re:MapReduce for C: Run Native Code in Hadoop by choppaface on Wednesday February 18, 2015 @08:01PM (#9071893)
Hmmm, this is definitely not some sort of vanilla C interface for Hadoop. MR4C is rather a ~40ksloc C++/Java library capable of performing somewhat high-level operations on datasets. It's closer to a MapReduce app or a tool like Pig than a simple C/C++ wrapper. In particular, MR4C has its own data model:

* https://github.com/google/mr4c/tree/master/tutorial/UserGuid...

... and nontrivial logging, traceback debugging, tempfile handling.. stuff for writing a library in MR4C

* https://github.com/google/mr4c/blob/master/native/src/cpp/im...

* https://github.com/google/mr4c/blob/master/native/src/cpp/im...

* https://github.com/google/mr4c/blob/master/native/src/cpp/im...

... and a 3600 line C++ package for extracting crops from satellite images based upon geospatial coordinates (that is likely more relevant to Skybox than Hadoop users)

* https://github.com/google/mr4c/tree/master/geospatial

... and singletons all over the place! (Kenton Varda would not be happy)

* https://github.com/google/mr4c/blob/master/native/src/cpp/ap...

Props to the Skybox devs for getting this out. All of this code almost certainly got thrown away as part of Skybox's transition onto the Google stack.

Have not yet found any benchmarks or illustrations of I/O savings.

Re:MapReduce for C: Run Native Code in Hadoop by shmerl on Wednesday February 18, 2015 @07:08PM (#9071728)
Hadoop also allows using different languages with pipes (https://wiki.apache.org/hadoop/C++WordCount), thought it's not really a fully native approach anyway. And there are various limitations in that method in comparison with Java APIs.

Google really had to open source their original map-reduce framework which was written in C++. It would have prevented Hadoop from being mostly Java based.

Re:MapReduce for C: Run Native Code in Hadoop by oxplot on Thursday February 19, 2015 @01:34AM (#9072755)
It's highly likely that their map-reduce framework is tightly integrated with and dependent on their own infrastructure (e.g. GFS, BigTable, etc).
Re:MapReduce for C: Run Native Code in Hadoop by shmerl on Thursday February 19, 2015 @02:43AM (#9072910)
That may be, but they could open those too ;) For example HPCC (https://en.wikipedia.org/wiki/HPCC) was opened by LexisNexis with all its distributed filesystems and etc. They kind of realized that such frameworks only benefit from being open.
Re:MapReduce for C: Run Native Code in Hadoop by maxlybbert on Thursday February 19, 2015 @11:04AM (#9074730)
They did open source Sawzall ( https://code.google.com/p/szl/ ), but the implementation only runs on a single machine.
Re:MapReduce for C: Run Native Code in Hadoop by shmerl on Thursday February 19, 2015 @02:59PM (#9076564)
Is there any reason for them not to open the whole stack?
Re:MapReduce for C: Run Native Code in Hadoop by a1k0n on Wednesday February 18, 2015 @06:14PM (#9071540)
I'm really confused. It appears to have no HDFS support? So how do you get data in and out of the cluster? I guess the InputFormat just takes care of that magically?

Also the code isn't even close to Google style. I guess that's not so surprising, come to think of it.

This is really pretty interesting. I've wanted to be able to do this for a while, and marshaling everything through the JNI I guess is the easiest way to do it.

Re:MapReduce for C: Run Native Code in Hadoop by quadrature on Wednesday February 18, 2015 @07:18PM (#9071763)
Im equally confused, they mention that data sources can be specified through hdfs uris.

I can tell you one thing though, JNI is never the answer.