Really the network is the least expensive part. To get IO that can saturate a 10G costs around 20k, while the infiniband card sets you back less than 4k. Now you're talking about 100G, which will go faster, you could easily be looking at a 500k box of ssds.

Wait, what? 10G is 1.2 GByte/s, you can get that from a single SSD easily. 100G is 12 GByte/s, so 5 consumer-level ssds. Squeezing 12 GByte/s over the buses twice might be tricky, but certainly not a 500k problem.

>To get IO that can saturate a 10G costs around 20k

Assuming 10G Ethernet is 8b/10b like a lot of other protocols, that's 1GB/s over 10G Ethernet.

Here's a $130 SSD with 500GB of storage: https://www.amazon.com/Mushkin-PILOT-500GB-Internal-MKNSSDPL...

That's 2600 MB/s read speeds. Or more than double your 10G Ethernet.

------------

RAID0 8 of them together with ASRock's M2. Quad Ultra, and you've got $1040 of SSDs + $200 for the 2x Quad Ultra cards, or just $1200 for 20GB/s read/write speeds. More than enough to saturate any network I'm aware of.

In fact, someone has already done this: http://www.guru3d.com/news-story/eight-nvme-m2-ssds-in-raid-...

They used a higher-end NVM.e SSD and measured 28GB/s (that's capital B, gigaBYTES) on the Threadripper + x399 motherboard.

10G is 64b/66b, so somewhat more efficient -> 1.2 GB/s Too fast for single SATA, but any nvme/pcie ssd should give that sustained.

10 gigabit Ethernet is 10 gigabits of data. It's mainly Infinband that used a horrible marketing tactic of saying the signal rate instead of the data rate.

PCIe 2.0, USB 2.0, SATA / AHCI, and more protocols are 8b/10b. So all of these protocols were 10-bits per byte.

Modern protocols tend to be 64b/66b or better. So that's why I listed "assuming 8b/10b", its hard to memorize which protocols are which.

Apparently I'm wrong. 10G Ethernet seems to be a more modern 64b/66b in any case.

10 gigabit ethernet is not 1 GB/s. It's 10 gigabits of data per second, or 1.25 gigabytes per second. The encoding is not an issue with these data rate numbers because Ethernet quotes their data rate as a data rate.