News for hackers. Stuff that matters.
Re:Computing shortest distances incrementally with Spark by bitL on Tuesday June 16, 2015 @12:35PM (#9727036)
Yikes, I had to stop when the author mentioned MapReduce for computing shortest distance. That's one of the problems for which MapReduce approach is extraordinarily bad.
Re:Computing shortest distances incrementally with Spark by taeric on Tuesday June 16, 2015 @01:07PM (#9727274)
Sadly.... I'm noticing that folks that have done one MapReduce job are convinced it is the way to go for all jobs. It is scary just how fad based development can be.
Re:Computing shortest distances incrementally with Spark by dimi-31 on Tuesday June 16, 2015 @01:11PM (#9727303)
To be fair, abstracting that fact away, the rest of the post is still an interesting thought experiment.
Re:Computing shortest distances incrementally with Spark by ignoramous on Tuesday June 16, 2015 @05:35PM (#9728850)
A better approach at the scale being discussed in the article would be to...?
Re:Computing shortest distances incrementally with Spark by karussell on Wednesday June 17, 2015 @03:51AM (#9730495)
What does 'scale' mean :) ? Solving graph problems are best done in-memory with a big machine (RAM!), otherwise you are an order of magnitude slower (at least) if you try to distribute. But if you really have no choice I would have a look if some of the graph databases has a good distributed model. My gut feeling tells me that even a bad approach there is faster than spark ...
Re:Computing shortest distances incrementally with Spark by ignoramous on Wednesday June 17, 2015 @11:12PM (#9736646)
I guess Titan DB would fit the bill: http://thinkaurelius.github.io/titan/

Facebook has blogged about scaling Apache Giraph to insane number of Vertices/Edges.