News for hackers. Stuff that matters.

A comment on Computing shortest distances incrementally with Spark.

Re:Computing shortest distances incrementally with Spark by ddrum001 on Tuesday June 16, 2015 @03:20PM (#9728164)
I don't think you have a choice, you'll have to use MapReduce if the data is too big to fit into memory. I believe that's what Facebook and Google do:

https://www.facebook.com/notes/facebook-engineering/scaling-...

http://googleresearch.blogspot.com/2009/06/large-scale-graph...

[ Parent | on HN ]
Re:Computing shortest distances incrementally with Spark by bitL on Tuesday June 16, 2015 @06:59PM (#9729217)
Your 2nd link describes Pregel, Google's distributed graph database specifically built for these kinds of tasks.

They were using Map Reduce prior to that, but it was a cascading mess.

[ Parent | on HN ]