So people are also refusing to use the PS4 camera, their cell phones, any laptop with a webcam, any tablet, the Moto X with its always-on microphone, etc? Because they could be used to infringe on privacy?

It's about risk management. What's the potential harm, what's the likelihood of that harm, and what would be the impact if this were to be used maliciously? It sounds to me like people are rejecting this because a) Microsoft and b) rumors and speculation.