Yeah, I've been doing exactly what you describe and it gets the job done. I just daydream of being able to press a button and all my peripherals are switched without significant lag.
Changing the input on my monitor requires three button presses, and comes with a noticeable lag. Then I've got to press a button on the bottom of my mouse, and another button for my keyboard. If I bought a fancy KVM, I'd be able to swap everything in a couple seconds! It'd be nice also to hook up a mic and webcam but I'll probably just think about it for another 6 months and never pull the trigger.
For speakers you can just use a Y connector. Add a wireless mouse with multiple outputs built-in (eg Logitech MX Master), or a separate gaming mouse if you prefer, and that just leaves the keyboard to switch between the two devices.