There's a difference between the "always on" communication these devices have and communication the user specifically requests.
When I want to make an airline reservation, I'm requesting the device to send the booking information to the airline. I'm not asking it to send a recording to the mothership of everything that happened in my home for the last 5 hours, which a human assistant would never do.
You have to be able to trust that Echo isn't recording everything you say, unless you prefix it with "Alexa", and that this behavior will never change (say this is the behavior for the average user, but with a police warrant, they're able to tap your Echo).
I'm part of the group that thinks the tradeoff is worth it for the convenience, but I understand why many people would disagree.
Human personal assistants were connected to the outside world -- how else would they make appointments and reservations, book flights, find out what the weather would be, etc.? The whole point is to be connected to the outside world, automatic or no.