There's no analogue in the Infiniband world to dirt cheap 1GbE RJ-45 switches though.
And both price tags will make Elon's "someone's scamming me with a 'you're an enterprise customer' surcharge" sense tingle. The price tags for anything enterprise networking related are seriously inflated, and I would not be surprised if just making your own NICs and switches is cheaper once you hit a certain deployment size.
I'm having trouble feeding things at 400GB/s (not a typo, it's gigabyte/s) per H100 box.
For 10 boxes ideally you want 4TB/s...
> > an IB switch is in the same ballpark as an ethernet switch with the same port speed
> And both price tags will make Elon's "someone's scamming me with a 'you're an enterprise customer' surcharge" sense tingle.
In the context of Tesla doing their own protocol and not-high-end NICs.
The hardest things would've been the DDR4 and PCIE interface. But as they're using standard interfaces, and last generation. I'm sure they got a good discount on all that IP and it didn't cost them hardly any man hours to integrate. And Tesla might've even already had the licenses and IP setup as they make other ASICs.
I didn't do a budget or anything, but at even 10Ks of units, I could see how this could save money. Or at least not loose money. Assuming a comparable IB network card is ~$1000, which I also didn't price.
And there could be other potential cost offsetting features, like power savings.
I mean yeah, but thats why you have negotiators. List price is what suckers pay.
As soon as you start to buy in job lots, or the total price comes to >$500k then stuff becomes a lot cheaper all of a sudden (within reason)
Having said that Infiniband is an arse to deploy, but not as much as custom networking protocol on custom silicon.
> Ideas you’ll never hear at Google or meta
You'd be surprised. Google has a very strong tradition of "not-invented here" which extends to some of our production networking gear as well.
To be fair, at the time, some of this was justified because the available devices on the market couldn't support our use cases back then.
Per section 3.2 of the 2013 B4 paper [0]:
Even so, the main reason we chose to build our own hardware
was that no existing platform could support an SDN deployment,
i.e., one that could export low-level control over switch forwarding
behavior. Any extra costs from using custom switch hardware are
more than repaid by the efficiency gains available from supporting
novel services such as centralized TE.
Adding to sibling comment about Google, Meta[1] built 2 large-scale production training clusters for science: one with Infiniband, the other one with a custom RDMA over RoCE fabric.
> Custom designing much of our own hardware, software, and network fabrics allows us to optimize the end-to-end experience for our AI researchers while ensuring our data centers operate efficiently.
> With this in mind, we built one cluster with a remote direct memory access (RDMA) over converged Ethernet (RoCE) network fabric solution based on the Arista 7800 with Wedge400 and Minipack2 OCP rack switches.
Google, Meta and Netflix are among the most obsessive on optimizing their infrastructure - it's bold to assume they haven't looked at their COTS network gear and thought "hmmm..."
1. https://engineering.fb.com/2024/03/12/data-center-engineerin...
There's also other differences, such as port counts. AFAICT Spectrum switches at 400Gbps have up to 128 ports whereas equivalent Infiniband NDR Quantum only have 64 [0].
When building clusters of 32K+ GPUs the network cost, power, transceivers etc start to add up.
[0] https://www.semianalysis.com/p/100000-h100-clusters-power-ne...
With the SN5600 for Ethernet (Spectrum-X), which is 64 physical ports, you're running each port at 8x100G-PAM4).
Usually, this is line-rate, but if the other side is slow for whatever reason (say the consumer is not draining data), you wouldn't want the sender to continue sending data.
If you also have N hosts sending data to 1 host, you would need some way of distributing the bandwidth among the N hosts. That's another scenario where the credit system comes. Think of it as an admission control for packets so as to guarantee that no packets are lost. Congestion control is a looser form of admission control that tolerates lossy networks, by retransmitting packets should they be lost.
Or I guess even RoCE
Are there other technologies they could have used?
Also, the 80us is supposed to be the worst case, where typical is supposed to be <10us. Again not knowing anything about infiniband, what’s the typical perf? I tried to google but the people who are talking about it are in the know in ways I’m not.
Thanks!
It is definitely possible to go much lower than 80usec on Ethernet. But obviously it depends on the scale, utilisation etc.
At the sizes of GPU clusters we're talking about these days - 32K and up - things get tricky.
The main alternative to Infiniband used in the industry is RoCE - Meta has written a lot about it [0].
There's several reasons to avoid Infiniband, such as cost, availability, vendor lock in, lack of experience etc.
Those are some of the reasons why many players are trying hard to make Ethernet work, see Ultra Ethernet [1].
[0] https://engineering.fb.com/2024/08/05/data-center-engineerin...
IIRC, Arista started off focusing on the financial market with low latency.
There's fairly well regarded in a general sense nowadays (at least /r/networking often has folks recommending them as a vendor).
The graph at the end shows they measured (one way) latency at 1.3 microseconds (compared with 2.0 for IB).
But for AI training, where you're simply shuffling around large stacks of matrices, my guess is latency constraints weaken.