Have your tried Synergy for your mouse/keyboard situation? Another possibility is perhaps a KVM switch that if you plug in two outputs into the same computer that it is recognized as two distinct USB devices. I don't know if it's possible but then you could pass through one to the VM directly.