Re:The Cost of Scalability in Graph Processing
by revelation on Friday January 16, 2015 @08:33PM
(#8902928)
Of course networks are always going to be orders of magnitude slower. That's because we're already at the very point where sheer distance becomes a limiting factor to performance.
Remember Grace Hopper:
After skimming the article, I’m thinking the real bottleneck here is latency since it mentions Hilbert curves. Currently networks are orders of magnitude slower than memory but that won’t always be the case. A big game changer is going to be content-addressable memory because we won’t have to worry about network topology as much. It will work more like BitTorrent and locally cache frequently-used data as needed.
Going forward, I have to admit that I’m not hugely fond of big data schemes as they’re currently conceived. There is way too much emphasis on using strange new databases and commodity hardware. I want just the opposite approach - low level access to data with a language like Go or Rust and new hardware with hundreds or thousands of cores on the same chip so we can get revolutionary performance (like with Bitcoin ASICs). Then if we want to double performance, we simple double the number of cores rather than hand-optimizing code, and that is going to be huge for productivity.