Re:One trillion edges: graph processing at Facebook scale [pdf]
by anotherangrydev on Wednesday August 12, 2015 @11:44PM
(#10052147)
Their test machines have 16-cores. They're running 2.9Gops/core. Plausible, yeah. I'm not used to see real things achieve almost 100% efficiency, sounds really good.
[ Parent
| on HN ]
Re:One trillion edges: graph processing at Facebook scale [pdf]
by cbsmith on Thursday August 13, 2015 @12:10AM
(#10052216)
To provide perhaps more perspective, here's a paper using 2004 technology: http://www.leonidzhukov.net/papers/ParallelPageRank-2004.pdf
With 70 machines & 140 processors, they did pagerank over 6.6 billion edges in 35 minutes. Using 285% the machines, you'd expect them to be able to do it in 12.25 minutes. Moore's Law should put each machine at 160x more powerful, and with 300x the edges they are finishing in 19 minutes. So we've improved by 2x over an 11 year-old software stack that we've long since abandoned for better approaches, using a hardware stack that is much more optimized for SIMD computations than the 11 year-old hardware.
Re:One trillion edges: graph processing at Facebook scale [pdf]
by anotherangrydev on Thursday August 13, 2015 @12:21AM
(#10052238)
I'm reading that one, thanks.
I really dont know the PageRank algorithm, but I was really impressed by their results, a trillion of "something" in 3 minutes. I thought that just reading things should take you a lot more than that but apparently not.
Really nice stuff.
Re:One trillion edges: graph processing at Facebook scale [pdf]
by cbsmith on Wednesday August 12, 2015 @11:56PM
(#10052180)
Data parallel problems are essentially the use case that should allow you to get to 100% efficiency. The hardware/software just aren't going to get it easier than that. In this case, I'd speculate that you'd not even need 50% efficiency to get this result.