For me switching playback device is quick and reliable if the device is a PC or phone. Whenever I try to select a connected Google speaker... it takes like 3-5 tries. I've found it is more reliable to use voice commands to get the speaker to play Spotify then I can use a phone/PC to select the song/playlist/podcast/etc. but it's kind of frustrating it just doesn't work well to begin with.