I think it was a buggy driver more than a Windows problem, as I tried selecting it as the default. It sometimes wouldn't show up when re-plugged in, as well. I'd even have to reboot sometimes to get it to show up as an option.
Doesn't work for me (not the person you are replying to) - every time I change input on my monitors Windows starts trying to use them for audio - doesn't matter what default I set, even disabling them.
Need to select it as default when plugged and whenever it gets replugged it retake its priority. Works as expected with multiple default / temporarily plugged devices.
Interface might need a bit of a clean up though.