It's a really interesting point. I've had to go through quite a bit of trouble to get non-keyboard HID devices (like Yubikeys) "un-Dvoraked" so that they work as expected.