That's interesting. I've had no problems with USB MIDI adapters. As long as I have less than 5ms latency in my audio path, I'm able to play comfortably over the cheapest USB-MIDI adapter on Amazon. I've also had reasonable success with USB audio interfaces, with latency around 5ms (IIRC - I switched back to my ancient emu10k1 card because of the DSP).
I still use a USB 2.0 MIDI controller and the latency is absolutely killer. Press a key, a beat later, see the note appear on piano roll...