For example I don't think reminders has geofencing at all so you're asking for something it just can't do. If you are just asking it to remind you of something at a certain time it will just do it no manual interaction required.
For the bathroom thing there's one leap that makes it tough, Google doesn't know if a place has a public restroom, at least it's not something they surface on Maps if they do, so to answer that it'd have to have had a manual thing added saying gas stations, rest stops, etc. generally have public toilets.
Are you on vibrate with no alert or media volume? Mine plays a little boop boop noise when it's initialized the mic and is ready for me to speak...
RE: Bathrooms - You hit it mostly on the head. I just wanted to find a gas station, fast food, restaurant, typically anything that would have public washroom access. I had no idea how far until the next town, what services they had, etc.
RE: Voice promtping. I had full bluetooth media volume (I always drive listening to podcasts) but virtually never have notification sounds on.
---
I think I do understand why there's limitations. But it gets to the point where if every use-case I have for a voice assistant is "we're just not there yet" then at a certain point that voice assistant is useless.
Microsoft Word had voice typing in like 2003 or some shit. If GA is only really good at doing Google searches or typing my voice in a text it honestly isn't that impressive of a feature to me; its gimmicky and not "flagship feature" imo.
EDIT: Another example. One of the very first things I tried doing with GA on my Pixel 1 was "What are the highway conditions?" as I was preparing for work. The google result I got was for something like Arizona highway conditions... even though I live in Sask. Canada.
As far as reminders and stuff go I'd recommend just trying that sometime when you're not driving just to see what the workflow is like. Sometimes you just have to set something up the first time you use a feature (like picking which calendar you want reminders saved in) and then from that point on it's transparent.
My issues in general which lead me to locking down the permissions, and not really looking back, is that its frustrating to work with. If I'm trying to text hands free in my car it has random delays that have no audio cue to work with. If I'm holding the phone, I can feel its vibrations telling me when its ready... but I'm driving, so handling the phone itself seems like a poor choice.
After that I tried some things like "Ok Google, when I get home remind me to put the trash out"... to which point it launches a calendar app and requires me to manually press things... or mishears me and just tries googling for random shit. Again, not helpful while driving.
I tried playing with it. I had my navigation open while commuting in an unfamiliar corner of the country once and I said: "Ok Google, I need to use the bathroom"... so it google searched constipation cures.
For the record, the permission it wants me to enable is Web & App Activity Tracking... which should not block me from voice texting but here we are.
---
Nitpickiness aside, it just isn't that special anymore is it? I don't see what's so special about a niche, buggy voice prompt that requires me to run apps manually half the time. Its a nice feature to include, but it has never felt like a "selling point" to the phone lines.