that's actually an optimization it seems, at the processor level there is a low power keyword listener for "Ok, Google" but it just fires the app it doesn't get anything more since you didn't grant it mic access.

A much crazier permission is phone access on Android. I recently noticed Facebook's Messenger had been uploading my call logs without ever mentioning it to me. Only realized when it recommended I call a contact back despite me not using messenger to call them - it saw the call in the phone's history and matched it against its database.