I use my chromecast to turn my TV on or off with voice commands, even when I just want to watch FTA channels.
So you use your device as a universal remote control. That is not at all the common use case.
I mostly use it so that a 'goodnight' voice command will turn off the TV, no matter what source is selected or what is playing.
That seems to be the main use case, but instead the design forces every source to be a universal remote control.