News for hackers. Stuff that matters.
Re:Bigger data; same laptop by e12e on Thursday February 05, 2015 @03:12PM (#9005673)
> If you think of the network as part of the memory hierarchy, transferring data to nodes in a distributed system is usually slower than even spinning disk storage.

According to Google quick search, the bandwidth of DDR3 is ~12.8 GB/s, or ~104 gbps. As I understand it second-hand (read: cheap) infiniband givies ~40 gpbs, and newer, high-end gives ~100gbps[1]. Now, if you're using regular gigabit ethernet (or even 10gps with cheap-ish switches) -- you'd probably see around 5-8 gbps -- or if we're being very generous, on the order of 1/10th of cpu-ram bandwidth.

Sata2 is rated at ~6 gbps for comparison. In other words, 10gpbs ethernet/infiniband should be on-par with local SSDs, but "proper" infiniband should push the node-node ram-ram performance up compared to that. It appears fiberchannel is also moving towards the ~100 gps mark[2], but isn't there quite yet. And you'd have to have something on the other end -- with high speed networking it is quite natural to assume that both source and destination is RAM.

It's actually pretty crazy with ~40gps infiniband -- I'm considering getting some second-hand gear just to play with at home. Seems both cheaper and more reliable to work with than try to get 10gps ethernet to work (on a home/hobby budget!).

Eg:

http://www.ebay.com/itm/Flextronics-F-X430066-8-Port-SDR-Inf...

Or:

http://www.ebay.com/itm/Mellanox-MHGA28-XTC-InfiniHost-III-E...

All that said, I think the article makes a lot of great points -- and it does seem a bit funny that people seemingly are thinking of 1 gbps as "fast" when so much has happened on the cpu/ram side since that was remotely true.

[1] http://www.hpcwire.com/2012/06/18/mellanox_cracks_100_gbps_w...

[2] http://www.pcworld.com/article/2096980/fibre-channel-will-co...

Re:Bigger data; same laptop by bane on Thursday February 05, 2015 @03:52PM (#9005996)
Good points.

Though, you'd be surprised at how often people get stuck on distributed problems where they just can't seem to get their cluster to process more than 80-100MB/sec and it turns out it's because they ran 1Gb ethernet between their nodes.

Re:Bigger data; same laptop by vardump on Thursday February 05, 2015 @03:21PM (#9005742)
And how about latency? DDR3 has 100 ns latency, give or take.
Re:Bigger data; same laptop by e12e on Thursday February 05, 2015 @03:42PM (#9005909)
If I'm reading the numbers right[1], and converting correctly, infiniband is on the order of 5000 - 500 ns. I didn't mean to imply that the network isn't slower, just that it's possible to do quite a lot better than gigabit ethernet quite cheaply.

[1] http://en.wikipedia.org/wiki/InfiniBand

Re:Bigger data; same laptop by vardump on Thursday February 05, 2015 @06:18PM (#9006828)
Yup. Infiniband + RDMA is pretty cool, but RDMA is also rather scary. I mean, writing remotely to another systems (physical) memory. In the same time I do wonder why this is not being used more nowadays.