Unless we're talking about an electret capsule with a physical LED wired into the supplying power rail that is switched off when the mic is not in use, you have to trust software.
And good luck with that after the patriot act. I am not implying the NSA has a microphone backdoor, but if they had and someone abused it, how would you know about it?
Listening in for keywords and only send text/audio when keywords are spoken isn't only good for ads, that would be dream of any intelligence agency. And since Snowden have been a few years.
"Citizen four" by Edward Snowden.
Those chips do not log or transmit the audio they have access to.
edit: Having re-read my comment, I can see how it could easily be read to say "It's happening and this is how it works", whereas I intended to convey something like "It could easily be done and here's how." I have a bad habit of implying my point rather than stating it outright. I'm working on it!
If it were happening, it would kill our battery really quickly, and be easily measurable and obvious in a router trace, so I find it very unlikely.
In this situation, I was alerted to this because the assistant started responding. However, I've also been in situations where I tried deliberately to talk to the assistant and it failed silently. In those situations, the UI spawns the Assistant interaction dialog, listens to what I say and then just silently closes. Sometimes this happens if there's too much background noise, for instance, and it then just re-evaluates that it wasn't a valid query at all and exits. Sometimes some background process may be frozen. Who knows if this happens before or after sending the data to the server. Sometimes the dialog lingers, waiting for the next input, and sometimes it just shuts off, leaving me (annoyingly) to have to reopen the dialog.
Putting that together, I have no idea how many times the Google Assistant has activated in my pocket, gone live, recorded stuff, sent it to Google's servers, realized it wasn't a valid query, and shut off without alerting me. I've certainly seen the Assistant dialog randomly open when looking at my phone plenty of times, which is usually a good indicator that such a thing has happened. If it silently fails in such a way that the UI doesn't respawn, then I would have no idea at all.
The net effect is that Google gets a random sample from billions of random conversations from millions of people every time this thing unintentionally goes off. They have a clear explanation as to why they got it and why ads are being served in response afterward. They can even make the case that the system is functioning as intended - after all, it'd be unreasonable to expect no false positives, or program bugs, or whatever, right? They can even say it's the user's fault and that they need to tune the voice model better.
Regardless, none of this changes the net result, which is they get a random sample of your conversation from time to time and are allowed to do whatever with it that they would have done if you sent it on purpose.
Have you tried resisting an export from Google Takeout to see if there are answers in that data?
It's much better and safer to assume they are than they're not, especially because I've seen many many results which indicate they are.
That said, there's a much easier way to test this. Take two identical voice recognition smart devices (think Amazon Echos), register them each with a new never used Amazon account. Modify one of the devices to have a switch on its mic input which you leave off, and one which you leave on. See if the one with the mic on starts showing ads for things you've never searched for on that Amazon account. If the other one doesn't then there's your answer.
That sounds interesting enough that I might just give it a try.
Lots of people run network traffic sniffers to see what apps are doing. Lots of people decompile apps. Lots of people at companies leak details of bad things they are doing.
Why has nobody been able to demonstrate this beyond anecdotes about talking about swimming pools and then getting adverts for swimming pool stuff?