I would love to have an always-on always-listening home assistant which did not ship my audio stream off to a cloud I don't control.

The fact that all the current listening assistants do ship the audio stream to the cloud has nothing to do with CPU power and everything to do with a desire to collect as much personal data as possible, and as far as I'm concerned they can pound sand.