Social Graph Analysis using Elastic MapReduce and PyPy

Re:Social Graph Analysis using Elastic MapReduce and PyPy by mcroydon on Wednesday May 04, 2011 @12:50PM (#2514393)

Thanks for the tip, I didn't think to copy the data to ephemeral storage like that. That'll probably speed things up a lot.

I ended up splitting the data in to a relatively small number (~200) of ~30MB gzipped files in order to initially saturate the mappers and speed things up. If that's not necessary after moving to ephemeral storage that's fine by me!

[ Parent | on HN ]

Re:Social Graph Analysis using Elastic MapReduce and PyPy by yummyfajitas on Wednesday May 04, 2011 @01:29PM (#2514549)

It's not necessary once your files live on ephemeral storage, but it would be necessary if you want the distcp operation to be fast. But again, the s3 block filesystem will not have this problem.

[ Parent | on HN ]