Are you aware of the extraordinary good deal on huge kintex (K420T) nic-like dev boards on aliexpress?

There are boards with 4 sfp+, ones with 2 sfp+ and 2 QSFP+, and even one with 4 QSFP28 (and UltraScale+ XCVU9P)...

https://www.aliexpress.com/store/group/FPGA-DEV/620372_25030...

they sound like great targets for your work...

Yes, I am aware of those. However, the kintex PCIe interface is a bit of a pain as it has a TLP straddling mode that can't be disabled, so it will be some time before it's supported as it will require some significant reworking in the PCIe interface modules. I am planning on supporting straddling eventually as this will improve PCIe link utilization on the ultrascale and ultrascale plus parts. If someone wants to donate a board, I can look in to supporting it.

Interesting, I never heard of straddling, what is it supposed to achieve?

Stradding is an artifact of very wide interfaces. On the Ultrascale+ parts, the PCIe gen 3 x16 interface comes out as a 512 bit wide interface. Every cycle of the 250 MHz PCIe user clock transfers 64 bytes of data. The issue has to do with how packets are moved over this type of interface. If your packets are all a multiple of 64 bytes, no problem, you get 100% throughput. However, if your packets are NOT a multiple of 64 bytes in length, you have a problem. What byte lane do packets start and end in? The simplest implementation is to always start packets in byte lane 0. The interface logic for this is the simplest - the packets always start in the same place, so the fields always end up in the same place. However, if your packet is 65 bytes long, the utilization is horrible - it doesn't fit in one cycle, so you have to add an extra cycle for every packet, and bus utilization falls to 50% as you have 63 empty byte lanes after every packet.

Straddling is an attempt to mitigate this issue. Instead of only staring packets in lane 0, the interface is adjusted to support starting packets in several places. Say, byte lanes 0 and 32. Or 0, 16, 32, and 48. Now, when you have a packet end in byte lane 0, you can start the next packet in the same clock cycle, but in byte lane 16 or 32. This increases the interface utilization. The trade-off is now the logic has to deal with parts of two packets in the same clock cycle, and it has to deal with multiple possible packet offsets.

The specific annoyance with PCIe packets is that the max payload size is usually 256 bytes, but every packet has a 12 or 16 byte TLP header attached, which really screws things up when combined with the small max payload size.

Fantastic explanation, thanks.

No interest in 40GB phy?

Right now, 40GB is the sweet spot in lower cost surplus hardware: E.g. you can get Arista DCS-7050QX-32 for about $500 shipped on ebay all day long.

100GB/25GB switches are still really expensive.

No free 40G MAC/PHY. Unfortunately, the Xilinx CMAC is 100G only, and the Xilinx soft 40G MAC/PHY is $$$$. I have looked in to building a 40G/100G switchable MAC/PHY, but it's going to be a serious pain in the rear.

Funny you mention that switch, we bought one of those off of eBay for our testbed as it supports PTP.

Also, for optical switching applications, one of the most important factors is how long it takes to bring up the link after switching. Because of this, we have no interest in spending time on 40G and 100G interfaces because interlace deskew takes hundreds of microseconds, and 100G also requires FEC which takes hundreds of microseconds to lock. So we're focused on 10G and 25G and running multiple links in parallel, which also provides more architectural flexibility. I added 100G support for three main reasons: the CMAC license is free, so why not?; supporting 100G makes the project a whole lot more interesting than only 10G or 25G, and it provides a simple way of testing the core NIC datapath.

oh... now I understand the purpose of the TDMA. Using actual optical switching to interconnect. Very interesting!

Got any pointers to the sort of optical switching components you're using?

[I've been out of the networking business professionally for almost a decade now, so I'm a bit out of touch with the state of the art in optical stuff--- I was somewhat surprised recently to learn of the existence and low cost of LR4 40gb optics. :P]

That's part of the research we're doing!

Take a look at: https://circuit-switching.sysnet.ucsd.edu/

And: https://arpa-e.energy.gov/sites/default/files/UCSD_Papen_ENL...

The current generation of switches that we're working on uses diffraction gratings patterned onto glass hard drive platters, installed in a modified hard drive, spun by a custom motor controller that's synchronized to the NICs via PTP.

Ha. I was going to guess it would be an AOM, I wouldn't have guessed a diffraction grating on a hard drive platter. That's awesome, and must be incredibly energy efficient.

The cost of switch ports and interconnects could all be dumped into making host interfaces faster, allowing for the switching time to be reduced.

What are the losses between ports that you are experiencing and what is the crosstalk between ports that are not connected?

Crosstalk is better than 30 dB, and double pass loss between ports is 5-8 dB. The switch is basically cycling through three or four different interconnection patterns that are defined by looped back fiber connections, so the signal has to pass through the switch twice.

Did you do a patent search ?

We have not done anything like that. Presumably that would be an important thing to do if a commercial product is produced at some point.