Many KVMs do not have a "raw" port like you mention. But manufacturers are becoming more enlightened.

The other thing is, many KVMs will add a hub between your keyboard/mouse and the system you're connected to. This is convenient because you have just one cable to each system. But sometimes this is a source of incompatibility. I think the hub requires the system to be able to enumerate devices one-level removed, and the enumeration might be the source of delay. It also might be beyond bios-level firmware, which is less capable than many OS usb stacks. I believe this might have been the problem with mac boot keys.

I know a mechanical usb switch is not very elegant, but I find the reliability it adds is worth it to me.