Ethernet tooling for HPC has a ways to go, but I suspect in the future it will be more competitive. Especially if specialty fabric vendors cut down on R&D.
CLOS fabric designs seem to be winning the war these days which I think favors Ethernet in the long run. Better flow distribution on aggregate links and now widespread support for MC-LAG means you can build a really wide CLOS network with L2-only.
That means that these switches, while fast, cannot check the packets for correctness (they don't have the full packet). That they will have "aborted" packets. That in some important ways these networks have the problems of the "half-duplex" networks of old.
Broadcom focuses on features for packet transmission. That means these Mellanox switches are pretty much restricted to situations where you want to have a set of servers on a single network segment and nothing else (not even an upstream connection). If that's exactly what you need, great. But mostly you're going to need more.
Your information may be old; Mellanox has pretty much the same feature set as Broadcom now.
Also, you can run cumulus on the switch, which is pretty awesome.
But yes, seems EVPN + VXLAN is the way the industry is going nowadays to build eth CLOS fabrics, whereas Trill & SPB seem more or less dead, for some reason.
Everybody is saying ethernet is simpler to manage than IB, but IME at least for HPC the opposite is true. IB is more or less plug and play, you get RDMA, multipathing etc. all right out the box. Whereas if you'd set up an equivalent thing with ethernet, you'd have to set up DCB, RoCEv2, EVPN+VXLAN+BGP (or something equivalent).
Intel is far worse to deal with, and additionally engages in anti-competitive architectural wars, preventing other vendors from interfacing with the CPU bus.
As a result we have NVLink, and now we will soon have official IB cards with NVLink ports, and probably ARM cores too.
Huh? Nvidia is by far the worst option when it comes to GPUs if you want to run Linux. Both Intel and AMD manage to have excellent open source drivers, while Nvidia's is a proprietary mess that everyone complains about and doesn't work all that great with typical distro update mechanisms.
IB is actually easier to reason about and debug than DCE, but obviously a different community of practice.
But I think this is the way the market is going in the longer term.
What I'm getting at is setting up clusters larger than what you can fit behind a single switch. So you'll want e.g. a CLOS fabric with multipathing (the typical IB setup, FWIW). As Trill and SPB seem pretty dead, it seems the momentum is to do the multipathing at the L3 level, using the aforementioned EPVN+VXLAN+BGP, or something similar.
Or maybe you could do one "provisioning and admin" VLAN that spans the entire cluster and which uses spanning tree, and then the high-performance RDMA stuff uses the per-leaf VLAN's and L3 multipath routing? Is that simpler and better performing that EVPN + VXLAN?
What is the routing latency on such BGP setups BTW? I find it hard to image you can get even close to eth (not to mention IB!) L2 latencies? Or can the fast paths be done in hw (or FPGA's)?
In most ASICs everything is the same latency since packets go through the whole pipeline whether they use all the functionality or not. Anyway, the latency of plain routing would have to be equal or faster than VXLAN encap + routing.
Their ethernet accelerator VMA stands for "Voltaire Messaging Accelerator".
Later Marvell went on to acquire Cavium for 6 billion dollars with aim to build an infrastructure company. Though from the company's latest earnings release it seems that the deal isn't really a good one.
Edit: sorry somehow glossed over your mention of OmniPath, didn't mean to restate what you already said.
Mellanox has apparently been under activist investor pressure to reduce their R&D expenses and pay more dividends. And then there was the rumors that Intel were interested, but apparently Nvidia in the end offered more.
From a HPC perspective I think it's good Nvidia got the deal, Intel is already a quite dominating force in that market, and if they'd have gotten the deal it wouldn't have surprised me if they would just have sunsetted it in favor of their own Omni-Path (which they could then develop at a leisurely pace due to lack of competition).
Though as I have mentioned before, I do wonder about the long-term prospects for Infiniband as a technology. Modern high-end ethernet does many of the same things with RDMA (RoCE), though I believe IB still has a latency advantage. And multipathing with ethernet is weird, seems both Trill and SPB are kind of dead, and most players seem to do multipathing at the L3 level (which might not be good for latency?). And in contrast to ethernet, IB is pretty much a single-player technology nowadays, so is the market big enough to bear the R&D costs to keep developing it?