I've seen this argument a lot, the issue is not people are not smart enough but people have different value toward privacy. I understand tech and yet I'm fine and even supporting this kind of tech. I understand its really hard and sucks to be in the side where you think is nightmare but other people think is not an issue. But thats the reality, people can have different value.
Second, the microphone on your smartphone is less powerful than the microphone on your Echo, and your smartphone isn't omnipresent in every room of your house simultaneously, including guest bedrooms.
Third, (opinion me) Amazon is a slightly less trustworthy company than Google, and a much less trustworthy company than Apple.
And finally, you can outright turn off assistants on your phone (and should). An always-on virtual assistant run by Google that always listens even when your phone is locked is also a privacy nightmare, but there's pretty much no reason to have that.
If you're paranoid about the security of carrying a microphone that could be enabled via software at any time, you can also install a custom ROM on your Android device that uninstalls Google Assistant and non-FOSS software entirely. LineageOS is really good, I highly recommend it -- it's just not feasible to install for regular, nontechnical consumers.
Again, how is this different from a smartphone that will wake up when you say "Hey Siri" ?
But, there are 2 perfectly capable home assistants on the market today that (right or wrong) aren't part of the same consumer-level narrative about privacy, and people like simple, straightforward, black-and-white narratives, like "Facebook cameras in your house are watching your child so that targeted ads can turn them into a nazi." That's a simple story, it's easy to remember and think about. Even ordinary, privacy-uneducated users like that story.
It's totally feasible to me that this device will flop due to privacy concerns. Scanning around quickly online, I can't see a single positive review (even from mainstream publications) that doesn't end with, "but of course Facebook is evil and you'd have to be stupid to trust them with a camera in your house."
I suspect Facebook just lost the narrative on this. They should have called it "Instagram Portal", and then the consumer situation would be completely different.
Most of my siblings have Alexa devices in their house. It's a privacy nightmare, but they don't care. Even so, none of them would be dumb enough to buy a Facebook portal. It's not even a sacrifice -- Alexa/Google work with more devices and have more features.