1. The ad company wants to target you. If your friend is talking the voice interpreter needs to be able to determine that it is not you talking. More than that, it would potentially need to be able to differentiate your voice from a multitude of voices at some social function.
2. Once your voice has been isolated then we're talking about natural language processing.
Admittedly, the technology required to do this is very close if it's not already here (GPT-3, etc.) but, to your point, it would at least not be a good look if Amazon (or Facebook, or Google, etc.) did this.
What reputation hit? The past 20 years of tech has shown that people in general don't care about privacy. They care about convenience.
I don't think reputation is really an issue for most major companies at all. People can't keep up on every sin committed by corporations and quickly forget.
Gerber and Beech-Nut have been found to sell baby food full of heavy metals. Literal poisons in dangerous amounts spoon fed into the mouths of our children, but they haven't gone out of business. DuPont knowingly poisoned people (and ultimately every last one of us) but somehow that doesn't matter. They're still operating and profiting. There are countless examples of companies who have done plainly evil things leading to deaths and suffering and yet they face zero meaningful consequences. Not from government, and not from people "voting with their wallet"
Companies only seem to care about their reputation because it can have an impact on profits in the short term which happens to be what many companies are overly focused on anyway. Don't want stock prices to drop even a little for a few months before everyone moves on to the next scandal.
All this talk about smartphones and Alexa listening to conversations to power ads fail to comprehend the economics of it. Constantly recording and analyzing audio to parse out relevant conversations would take lots of compute power and transmission usage. I am incredibly skeptical that this would yield enough of an advertising edge to make it a net positive. And this isn't even touching on the reputation hit of companies did this.