Social Graph Analysis using Elastic MapReduce and PyPy

Re:Social Graph Analysis using Elastic MapReduce and PyPy by yummyfajitas on Wednesday May 04, 2011 @12:05PM (#2514251)

It's usually more efficient to use distcp to copy the data to ephemeral storage:

    hadoop distcp s3n://<id>:<secret>@bucket/input_data /input_data
    hadoop jar myjar.jar org.myorganization.module.jobname /input_data /output_data
    ....other steps....
    hadoop distcp /output_data/ s3n://<id>:<secret>@bucket/output_data

Doing it this way, all spills/intermediate steps live on (very fast) ephemeral storage. Additionally, the distcp command is pretty fast since it is a distributed copy - all nodes will saturate their connections on the copy.

Also, the authors advice to "make processing faster, decompress it, split it in to lots of smaller files" is probably not the optimal way to do it. Hadoop isn't great at storing lots of small files.

http://www.cloudera.com/blog/2009/02/the-small-files-problem...

I'm guessing that he gets a speedup because he is using raw s3:// as a filesystem rather than s3n://. If I understand the hadoop s3 connection, this means that hadoop will use a single connection to pull a single file - so a single 10GB file will max out a single node's network connection and the rest will go unused. Splitting the file up into 1 million line blocks lets multiple nodes pull simultaneously. But storing files using the s3n:// allows this anyway, and results in large files being split into 64mb blocks. It also allows you to store files larger than 5GB, or whatever the s3 file size limit is.

Now, a question I'd really like to know the answer to. What advantage is there to using EMR vs Whirr?

[edit: adjusted language - upon rereading it seemed harsh,which was not my intent. ]

[ Parent | on HN ]

Re:Social Graph Analysis using Elastic MapReduce and PyPy by mcroydon on Wednesday May 04, 2011 @12:50PM (#2514393)

Thanks for the tip, I didn't think to copy the data to ephemeral storage like that. That'll probably speed things up a lot.

I ended up splitting the data in to a relatively small number (~200) of ~30MB gzipped files in order to initially saturate the mappers and speed things up. If that's not necessary after moving to ephemeral storage that's fine by me!

[ Parent | on HN ]

Re:Social Graph Analysis using Elastic MapReduce and PyPy by yummyfajitas on Wednesday May 04, 2011 @01:29PM (#2514549)

It's not necessary once your files live on ephemeral storage, but it would be necessary if you want the distcp operation to be fast. But again, the s3 block filesystem will not have this problem.

[ Parent | on HN ]