You're painting this like AMD is off to play in their own sandbox when it's more like the entire industry is trying to develop an alternative to Infiniband.

Ultra Ethernet is a joint project between dozens of companies organized under the Linux Foundation.

https://www.phoronix.com/news/Ultra-Ethernet-Consortium

>> The Linux Foundation has established the Ultra Ethernet Consortium "UED" as an industry-wide effort founded by AMD, Arista, Broadcom, Cisco, Eviden, HPE, Intel, Meta, and Microsoft for designing a new Ethernet-based communication stack architecture for high performance networking.

You probably can't call it "industry standard" yet but the goal is obviously for it to become one.

I wrote:

> It is weird to see the industry invent yet another standard to do effectively the same thing just because Nvidia is using it.

This is a misstep for all involved, AMD included. Even if AMD is following everyone else by jumping off a bridge, AMD is still jumping too.

You hardly beat someone by copying him. They have way more experience in the field you try to catch up.

AMD doesn't need to beat Nvidia, they just need to match them at a lower price point.

In business, that combination is nearly impossible to distinguish from beating them.

That‘s beating them at the price point.

You don’t beat someone by doing everything worse either.

Infiniband is a monopoly from NVidia (Mellanox). Everyone else would much rather use Ethernet which is the actual industry standard.

Depends on deployment: Ethernet has more drawbacks encapsulating PCIe packet traffic than Infiniband does, or lack thereof with RDMA.

Others can build infiniband hardware according to the standard. There used to be at least two companies building infiniband hardware until Intel killed QLogic’s infiniband division in a misguided attempt to make its own monopoly. :/

Or they can develop a RoCEv2 that works (which is basically what UltraEthernet is).

Infiniband is not fun - it's a special snowflake of an interconnect that sits parallel to the rest of your datacenter network, and can not really run a standard TCP/IP codebase (yeah IPoIB is a thing but still). Do the Nvidia boxes really need a scale-out IP network as well as an Infiniband network?

Plus the spec is old. Packet spraying and trimming, better ordering guarantees, queue pair scalability... a whole bunch of enhancements have been incorporated into UE all the while being compatible with regular Ethernet.

Qlogic was never really an Infiniband vendor --- their qib driver is still in the Linux codebase and essentially emulates verbs on top of a messaging-based design.

We must agree to disagree then. Infiniband is an awesome technology. It was originally meant to to serve as a fabric to connect all components in a computer. That plan died with the dotcom bubble as it was too ambitious and therefore risky, so it was relegated to networking where it was an incredibly good job in delivering messages reliably and fast. It still am hopeful someone will do that someone will do it.

Nvidia has a unique problem, wants to move fast and has a shit load of money.

No need for Nvidia to go first to an industry standard and neither for AMD.

Personally would be great its getting backported but its so far away from an normal use case.

Nvidia is the one who went to an industry standard before their competitors in this space. It was created in 1999 and is called infiniband:

https://en.wikipedia.org/wiki/InfiniBand

Infiniband is extremely popular in the HPC space, which is why Nvidia adopted it. Everyone else saw Nvidia adopt it and said "Let us make a new network standard to be incompatible". This is mind boggling.

Even more mind boggling is that many of the companies in the Ultra Ethernet Consortium are members of the Infiniband Trade Association, AMD included:

https://www.infinibandta.org/member-listing/

This would be like the automotive industry forming a consortium to invent new incompatible wheels to exclude a successful upstart that adopted their existing standard wheel designs. With trillions of dollars in revenue on the line, you would think that companies would use existing networking standards to focus on building competitive hardware with reduced time to market, yet they are instead reinventing networking standards just because they can. This is a huge gift to Nvidia, since it means that everyone else is wasting time and money instead of being competitive.

Sorry to hound you for the third time, but this is wrong:

> Infiniband is extremely popular in the HPC space

Not anymore. There used to be Cray Aries/GNI, psm/psm2, and now there's Slingshot, the new Cornelis stuff etc. There's almost no Infiniband now.

The top500 says otherwise:

https://www.infinibandta.org/infiniband-and-roce-advances-fu...

Where are you getting your information?

I concede that the specifics of what I said were wrong, but the larger point was not.

If you buy a single DGX H100 rack and run LINPACK, you automatically get TOP500-grade numbers. Infiniband is a solid product, if not the best commercial offering for AI/ML, but no one buys it for an HPC cluster separately from the DGX boxes.

#26 on the list uses AMD GPUs with infiniband:

https://www.top500.org/system/180171/

You can likely find more. Infiniband has been excellent for HPC since the 2000s. That includes all HPC workloads, not just AI/ML.

Excuse me if I do not believe your claims concerning infiniband. They contradict not only actual data, but also what I have heard from people I consider experts.

Also, you did not answer my question concerning the origin of your information. I notice from another comment if yours that you have been talking to a LLM about this conversation. Have you been posting things that a LLM tells you?

> Infiniband is an industry standard

Infiniband is not an industry standard lol.

Maybe it used to be, but it definitely is not anymore. Most Infiniband vendors are dead. The only product from those days that endures is Cornelis' Omnipath, and even that only emulated the Infiniband API back with its first gen, and then evolved to be its own thing.

At this point, Infiniband is as good as a proprietary interconnect only sold by Nvidia/Mellanox.

Infiniband is very much an industry standard:

https://www.infinibandta.org/member-listing/

As far as I know, everyone is free to sign up with the infiniband trade association and implement the specification.

Furthermore, if you use RDMA over Ethernet, you are using infiniband at a low level. RoCE which enables it was originally called Ethernet over infiniband. It is maintained by the infiniband trade association.

Omnipath was Intel’s failed effort to try to kill an open standard. It purchased QLogic’s infiniband business, killed it in favor of omnipath and sold it when it failed.

> Furthermore, if you use RDMA over Ethernet, you are using infiniband at a low level. RoCE which enables it was originally called Ethernet over infiniband.

This is wrong.

RDMA over Ethernet is... RDMA over Ethernet. There is no Infiniband involved.

RoCE was motivated by supporting RDMA, which was then an IB-only feature, over regular Ethernet. The user-level APIs are the same (verbs), but the underlying architecture is all different --- it is traditional Ethernet with link-level flow control to make it lossless (pause frames).

This is not what I have been told by others.

Tip: copy-paste our entire exchange as a Gemini prompt and ask "who's correct?".

(I did that yesterday and was very satisfied :) )

You are not the first person to say “the AI assistant liked what I said over what you said” to me on the Internet, to which I say they will tell you whatever you want to hear.

I am not interested in continuing this discussion, but if you want to do your own research, I suggest starting with the fact that the Infiniband Trade Association controls the RoCE specification. I suggest you avoid using LLMs, for obvious reasons.