Yeah, I think you are correct here. I think I would also be happy if someone else was selling me the hardware and I controlled it more, and it just was a hardware pipe to the voice service provider.

Hotword recognition does already exist and works on fairly modest hardware (I've never tested it on a pi, so I don't know specifically how well that would work - or more accurately how fast). The trouble is that it's an enormous hassle to set up and while it is open source, it involves some company or other having it's grubby paws all over the recognition networks. Which may or may not rub you the wrong way.