You are correct that the actual processors that are "always" listening are fairly limited (They describe it as a DSP, so that should give you an idea of its capabilities), but when layered the way they are in Pixel phones, it makes it possible for the DSP to detect "something" interesting is happening, then wake up and pass the information to an "AI processor" (they call it the "Pixel visual core" but i find that a misnomer because it's a pretty capable TPU on it's own) which would then do the transcribing, and go back to sleep.
I'm making some assumptions that I haven't researched myself, but I'm assuming there are patterns, frequencies, and other simply detected "signals" present in most speech that make it easy to detect "someone is speaking", and from there can wake up a more powerful (but still surprisingly efficient) processor to do more work.
And Google also has some simplified and compressed neural networks dedicated to transcribing speech which are designed to run on lower powered devices[2]. I don't know if they are actually using the PVC in the Pixel phones to do audio transcription or if they're just using the general CPU for that, but I have a feeling that a TPU would be able to scream through buffered audio very quickly and would go back to sleep very quickly leading to minimal power draw.
In fact I'd be willing to bet good money that an "always transcribing" system could be done today with current generation phones in a way that wouldn't impact battery life enough to be obviously noticeable or cause the user to start looking for what was draining it in most cases. And considering how often I'm personally actively speaking in a day, I bet it would have virtually no impact on my personal device if implemented this way.
(and man did it feel wrong to type that last paragraph out!)
[1] https://ai.google/research/pubs/pub46522
[2] https://ai.googleblog.com/2019/03/an-all-neural-on-device-sp...
Are there now co-processors running all the time that could do full speech recognition (not just keyword recognition, which is way easier, but full speech recognition) all the time without such significant battery drain that it would be noticed? I ask not because I consider this some sort of technical impossibility, but because if that is the case, I'd like to know so I can update my understanding of the situation.