> Because of the polling nature of USB, you've got to wait for the bus to ask for data, which means you're going to have at least a couple of milliseconds of latency.

That's up to the driver. USB 2.0 can poll at up to 8kHz. USB 3 is harder for me to find exact numbers on but it has more flexible timing and can go even faster.