Rosetta: The Engine Behind Cray’s Slingshot Exascale-Era Interconnect

Re:Rosetta: The Engine Behind Cray’s Slingshot Exascale-Era ... by wmf on Wednesday May 06, 2020 @01:02AM (#23088044)

When I was doing a private cloud last year these features were way more work than checkboxes, but maybe that's what we got for using OpenStack. I finished the planning for jumbo frames to the VM (this was complicated by double VXLAN), SR-IOV was blocked because we only had ConnectX-4 but SR-IOV over LAG requires ConnectX-5, and the storage team never got around to Ceph over RoCE.

[ Parent | on HN ]

Re:Rosetta: The Engine Behind Cray’s Slingshot Exascale-Era ... by jiggawatts on Wednesday May 06, 2020 @01:32AM (#23088146)

Most enterprises run VMware, Hyper-V, or XenServer. Almost always with Cisco Nexus 9K series switching. With that kit it's surprisingly easy, but not with the default Broadcom NICs, they generally don't have the required features. Even if they do, the advanced features often don't work properly.

I've seen jaw-dropping performance when it was set up correctly, something like 99% of the theoretical wire rate of 20 Gbps. With dual NICs and Windows 2016 we got nearly 40 Gbps because it can use both NICs at once! Amazing file copy times. I tested it with the usual 4GB ISO file copy test and it was so fast there wasn't even a progress bar!

Meanwhile, on most networks I see about 1.5-3.0 Gbps effective across dual 10 Gbps NICs. If you think about it, that's crazy bad. It means that in effect they're getting less than 10% of the available physical capacity for data traffic. The other 90% is being thrown away by "passive" secondary cables or by the lack of configuration on the switching layer.

[ Parent | on HN ]

Re:Rosetta: The Engine Behind Cray’s Slingshot Exascale-Era ... by exikyut on Wednesday May 06, 2020 @09:58AM (#23090970)

I discovered the other day that you can google `Xgb/Xgbps` and it'll give you a result in seconds. 4GB is 0.8 seconds :)

And regarding that 1.5-3.0Gbps across 2x10Gbps... a) wat?!?! and b) I wonder if the number of horribly misconfigured networks exactly like that are out there giving 10Gig a bad name and hurting adoption.

[ Parent | on HN ]

Re:Rosetta: The Engine Behind Cray’s Slingshot Exascale-Era ... by timeu on Wednesday May 06, 2020 @09:44AM (#23090827)

Yeah, I can feel your pain. We deployed a fully virtualized HPC cluster on top of OpenStack and there was a fair amount of testing and playing with knobs (SRIOV, Jumbo frames, CPU pinning, etc) to get close to bare metal performance on a 100 GBit/s SDN fabric.

[ Parent | on HN ]