Sadly I can't describe all actions I've done (I just don't remember) but one of the most crucial is to replace you home screen app.
Zim/Omega launcher is a good starting point - Zim doesn't know about assistant, Omega has the options to not to use it.
Along with Hacker's Keyboard, F-Droid/FoxyDroid and methodologically going through the Apps and Permissions in Settings helps to tame this Hydra.
do a few minutes of research in advance to buy a phone that has a prayer (Pixel 3a is popular in this thread),
and install Graphene on it from day 1...
I have the generic "Google" app disabled, which I think is the thing that listens on the microphone for that. I don't really know what else it does, so I'm not sure what else is unavailable once it's disabled. I don't expect I'll be missing much though.
There is, actually: Android One. E.g. Nokia makes phones that have minimal crapware preinstalled.
You will still have to deal with Google's own crap... like the retarded Assistant that just won't fucking die. Even with absolutely every setting in every google settings page in every google app explicitly disabling the Assistant, if you dare speak words resembling "ok google" it will still "helpfully" pop up asking if you'd like to enable it. And sometimes spontaneously. While you're typing, hoping you'll fat-finger "yes please".