Infiniband is pretty neat for a given use case but that cable length kinda killed it for many data center use cases.

Still, if folks can deal with those short distances there's lots of inexpensive equipment out there.

Point taken, but for some time Mellanox has touted at least cross-campus-type IB. I never persuaded anyone to evaluate it for use between two machine rooms, but there were obvious non-HPC cases for it within them. I'd be interested to know how the long distance stuff actually works, not being a beginner with sales talk.

That's a good question. I haven't used it but extending infinaband ... I'm guessing there's some sort of intermediary devices doing some sort of piplining or emulation, that's usually how that works out.