When I was doing a private cloud last year these features were way more work than checkboxes, but maybe that's what we got for using OpenStack. I finished the planning for jumbo frames to the VM (this was complicated by double VXLAN), SR-IOV was blocked because we only had ConnectX-4 but SR-IOV over LAG requires ConnectX-5, and the storage team never got around to Ceph over RoCE.

Most enterprises run VMware, Hyper-V, or XenServer. Almost always with Cisco Nexus 9K series switching. With that kit it's surprisingly easy, but not with the default Broadcom NICs, they generally don't have the required features. Even if they do, the advanced features often don't work properly.

I've seen jaw-dropping performance when it was set up correctly, something like 99% of the theoretical wire rate of 20 Gbps. With dual NICs and Windows 2016 we got nearly 40 Gbps because it can use both NICs at once! Amazing file copy times. I tested it with the usual 4GB ISO file copy test and it was so fast there wasn't even a progress bar!

Meanwhile, on most networks I see about 1.5-3.0 Gbps effective across dual 10 Gbps NICs. If you think about it, that's crazy bad. It means that in effect they're getting less than 10% of the available physical capacity for data traffic. The other 90% is being thrown away by "passive" secondary cables or by the lack of configuration on the switching layer.

I discovered the other day that you can google `Xgb/Xgbps` and it'll give you a result in seconds. 4GB is 0.8 seconds :)

And regarding that 1.5-3.0Gbps across 2x10Gbps... a) wat?!?! and b) I wonder if the number of horribly misconfigured networks exactly like that are out there giving 10Gig a bad name and hurting adoption.

Yeah, I can feel your pain. We deployed a fully virtualized HPC cluster on top of OpenStack and there was a fair amount of testing and playing with knobs (SRIOV, Jumbo frames, CPU pinning, etc) to get close to bare metal performance on a 100 GBit/s SDN fabric.

Managing flow control, misc switching changes, mtu is as much an art of trial and error as it is a science.

Unless you are in a very controlled environment it is VERY easy to actually slow down a network while trying to make it more efficient.

Indeed. Infiniband was even meant for "datacentre" use, and I have better experience of it in compute clusters than Ethernet apart from the performance.

Infiniband is pretty neat for a given use case but that cable length kinda killed it for many data center use cases.

Still, if folks can deal with those short distances there's lots of inexpensive equipment out there.

Point taken, but for some time Mellanox has touted at least cross-campus-type IB. I never persuaded anyone to evaluate it for use between two machine rooms, but there were obvious non-HPC cases for it within them. I'd be interested to know how the long distance stuff actually works, not being a beginner with sales talk.

That's a good question. I haven't used it but extending infinaband ... I'm guessing there's some sort of intermediary devices doing some sort of piplining or emulation, that's usually how that works out.

Yup, and quite probably no one wants to turn them on for fear of bugs and sudden disruptions. Besides, IT is a cost right? /s The rock-bottom cheap support company outsourcing the infra maintenance is more careful about minimizing actual work and matching the exact wording in the support contract and not a cent more