USB? USB 3.1 is 16X slower than PCIE and maybe 100X slower than DIMM. USB is not an OCP recommended peripherals interface, it's for consumer removable products.

Then again, I'm not entirely sure about the requirement for a high speed interface in AI. I'm assuming a fast interface is needed as the amount of data ingested is typically large

People use DIMM as a general purpose interface?

No, but it can be done. I worked on an FPGA board interfacing through the DRAM interface once: https://www.researchgate.net/publication/4139277_Pilchard_-_...