At least initially (~2005 or so?) we had customers who were given free IB for their HPC clusters ripping the IB out and replacing it with with paid Myrinet installations because they could never get their IB clusters to work reliably.

Although we had the superior product, we could not market our way out of a wet paper bag. Our CEO counted on our (then) superior engineering & expected our products to market themselves. Eventually the IB engineering caught up and surpassed ours, but our marketing never even came close to theirs, so they were the clear winner.

I build a myrinet HPC cluster or two, vague memory that it was 2.5Gbit. Worked well, I particularly liked the cables (thin and optical, way nicer than IB cables). When I went to replace them the Myrinet 10G was repeatedly delayed, and seemed to be focusing on 10G ethernet, not MPI/HPC.

We ended up going with a new upstart called Pathscale, which then was bought by qlogic, and then Intel. Our benchmarks showed a clear win over Mellanox at the time. In particular on larger jobs the Mellanox drivers seemed to consume a fair bit of ram for each client you were talking to, so even 32 nodes jobs would consume a fair bit of ram. We even built a cluster out of hypertransport connected infinipath cards in a "HTX" slow instead of PCIe. Managed I believe 1.4us latency, which is still reasonably competitive today. We built SDR (first public cluster), DDR (first public cluster), QDR based clusters. But then Intel said the new Infinipath would not be supported on AMD nodes, or even previous gen Intel nodes, I said it's a non starter and switched to Mellanox. I talked to several other HPC sites that said the same thing, months later Intel killed off the product. Sad and short sited on Intel's part.

Where'd the name Myricom/Myrinet come from? "Infiniband" is a pretty cool name.

unless I'm thinking of the wrong vendor, the myrinet devices supported programming the nic. for some applications that was a really big win over IB verbs

Yes and no. The original firmware / host driver stack was open (shared?) source called MyriAPI. It was so terrible that spawned an ecosystem of uni and research lab projects to implement replacements. I worked on one such project at Duke (Trapeze). Others included BIP, Fast Messages, Active Messages.

MyriAPI was later replaced by a new firmware / host stack called GM. I think at some point we stopped shipping firmware source. I think that point was the transition from MyriAPI to GM...

I think that's a confusing reference to the fact that InfiniBand initially was supposed to be also a replacement for system-internal buses (i.e. a PCI replacement), before PCIe took that place instead.

I wasn't there, but understood it was intended for general machine-room -- I dislike "datacentres" -- use; Wikipedia implies it was intended for ~everything...

Anyway, IB at least had lower latency than Myrinet, which is what counts in a lot of HPC. I don't remember the numbers, but I got quite close to Myrinet latency with the on-board 1GbE NICs on Sun x2200s, an un-managed switch, and Open-MX, which was interoperable with the Myrinet MX protocol. (I remember MX being slagged off by Mellanox; I don't know how similar MXM later was for IB.)

Yeah that makes sense, it's an RDMA protocol, and PCIe has wayyyy more in common with infiniband then it does with legacy PCI.

You can use a bus PHY in a p2p configuration.

But Infiniband didn't do that.

you can do point-to-point connections with infiniband, it's a pretty common homelab use-case since adapters are so cheap these days.

InfiniBand is always point-to-point, that's the point. The talk about "bus" is confusing/wrong since it never is a bus.

oh, "bus" in the context of an actual bus, like CAN. I thought p2p just meaning direct-attach. But yeah it's switched, it's not a shared bus.

Yeah that makes no sense.