I'd imagine that's for the same reason why you can't "listen" to a video on mobile with the app in the background: it would qualify as a music stream, and YT doesn't have the rights to stream music in an ad-based model.
I have airconnect on a raspberrypi so I don't have to keep my laptop on and I can play songs from my iphone that way, just stream to chromecast audio. It's also worth mentioning it works a bit better than mkchromecast but it depends on your wifi.
"By the way, it's easy to cast audio only from YouTube or any other app - just go to the Google Home app and cast your phone's audio to the device or group."
Small things but they matter depending on use..
I believe Firefox on Android may also support casting (along with ad blocking and an extension to play Youtube in the background by removing the page visibility api)
I did exactly this about 6 years ago, worked fine. Shame to cast all those video bits through your wifi contention just to throw them away at the adapter, though.
Basically, I can only use it with Google Music. I rarely can use it for other things because it will only cast a subset of things that support casting. For example, I can't cast YouTube to it, because it doesn't support video. But what if want everyone in the room to hear what I'm playing and the video is irrelevant?
I'm considering buying an HDMI->DVI/Audio adapter and just using a standard Chrome Cast.
I feel like Google had the right idea here. But just missed the mark.