IIRC the audio is the same for all formats, so this shouldn't be an issue. You just continue playing what you have and stop downloading new video chunks. I think the biggest problem is that when you switch to the tab you would need to wait a bit for the video to appear (or it would need to pause and stutter). That is actually quite a hard problem as well because the player needs to decide how far into the future it should start fetching video.

Google provides renditions in two codec pairs for most video - vp9 + Opus and h.264 + AAC. Assuming the manifest for audio only keeps you using the same codec family of renditions, this should work just fine (minus, as you said, the player not showing anything until the next video+audio segment is fetched) as long as we're talking VOD content. Live content should also be able to switch from source to audio only with no re-fetch, but live transcoded renditions won't be in line with either rendition, so that'll cause a burp.

I feel like the YouTube Music app is actually changing to a totally different SKU when you switch between video/audio, and not just giving you the audio stream from the video.