^^ this is huge. I was looking at CPU's for a ML recently and Intel is out of the question. Chips with enough lanes to hit full speed on 4 GPUs + SSD cost twice as much as AMD.

This may even apply to high end gaming machines. Soon as you have 2 SSD's or video cards you will exceed link budget on most Intel CPU's and everything slows down.

It also happens to routers. 10 gig NICs plus attached SSD storage, over link budget again.

Intel's stupid market segmentation is biting them in the rear

Not just 10Gbps (such as the Intel card which is four 10Gbps SFP ports in one slot), but single and dual 100GbE per slot like this:

https://www.intel.com/content/www/us/en/products/docs/networ...

In calculating the bandwidth and pci-e bus throughput needed, a single 100GbE port is full duplex, so one has to budget about 210Gbps per port.

The funny thing is that some of the best 100GbE NICs for x86-64 servers on the market right now are Intel, but are best used on an AMD platform...

Nah, PCIe is also full duplex you don't need to double it like that but Mellanox has dual 200gbe cards. Here's the PCIe 3.0 version: https://cdn11.bigcommerce.com/s-uxkkta8o/images/stencil/1280... and here's the PCIe 4.0 version: https://cdn11.bigcommerce.com/s-uxkkta8o/images/stencil/1280...

Yes, the 3.0 version needs two PCIe x16 slots.

What? Why do you think the Intel 100GbE NIC is good?

We've been quite happy with Mellanox and Chelsio 100GbE NICs. The latest from each can do in-line HW TLS offload, which is a killer feature for us. No Intel NIC can do that.

IMHO the last good Intel NIC was the 10GbE "ixgbe" NIC. The design of the NIC was so tight as to be almost beautiful.

Recent 40GbE (and 10GbE based on the 40GbE chipset), and the new 100GbE NIC have the feel of being designed by a committee with endless features of questionable value stuffed in and consuming power and chip area.

Primarily the state of its Linux driver, and rock solid support in VyOS (derived from Debian stable). For a router, TLS offload isn't so much of a consideration when the system isn't doing anything at layers 4-7 in the OSI model.

If I had to make a perhaps overly broad generalization, I see more Chelsio and Mellanox NICs used in end point servers, and more Intel used in DIY whitebox network equipment.

That's fair; I come at this from a CDN perspective where end system performance is most important.

Do you see any benefit from the fancy features? Can it source/sink min sized frames at 100GbE? (144Mpps) ?

Intel's E810 based NICs require only a PCIe3.1x16 slot. 16 lanes will accommodate the 100GbE port just fine. Theoretical PCIe throughput for 16 PCI lanes is around 252Gbps. The 800 NIC chipset is just four 25Gb Ethernet lanes stitched together. PCIe4 won't help this NIC much.

Your throughput calculation is based on a single port 100GbE NIC working okay at full duplex bidirectional line rate in a PCI-E 3.0 x16 slot, which is true. Let's say that we budget 220-230Gbps of throughput per optical transceiver, such as if you were to max out a 100GbE link both directions with iperf3 for testing. But the intel X820 card has two ports on it, so your bandwidth needs are going to be in the range of 500 Gbps.

also, cumulative number of pci-e 3.0 or 4.0 lanes in a system is a big consideration if you want to have, for instance, four dual-port 100GbE NICs all talking to one CPU. Or some mixture like three dual-port 100GbE NICs + one or two 4-port 10GbE NICs.

Where the lower end of the Intel server CPU offerings really falls flat is not having anything close to 64 or 128 PCIE lanes at a reasonable price.

The maximum throughput of Intel's Columbiaville NICs is 100G max. When the 2nd port is used the total bandwidth is split in half, so there will be two 50G links. The 2nd port is usually in stand-by mode when the 1st port is used as a 100G link. So, two active ports on a Intel 100G NIC does not result in 200G throughput.

This is correct. The Asics on these cards cannot do 200G, even if pcie allowed. The only one I'm aware of that can is the Mellanox cx6 dx

I don't understand why you are combining tx and RX. PCI, fiber, and the switch/nic are all full duplex, so you don't need to add the two directions together. A dual port nic, even with pcie4, is unlikely to be able to hit 200G ( single direction) unless it's using very large packet sizes.

Since you're limited by the 100G negotiated speed, you nic will never send more than 100G with iperf on a single port.

100% this. With NVMe and higher bandwidth NICs, fast and plentiful PCIe lanes are increasingly important and Intel have just been so far behind in this (or gated equivalance at hilarious price differentials from AMD - like having to go to $4k Intel parts (and beyond) to match AMD prosumer desktop parts).

Just try to attach a 8 GPU A100 unit to a 2P Intel system, then try to add fast Infiniband with NVMe for scratch, and whole platform just chokes.