This is silly. Do you also worry about everyone's phone? It is not uncommon to see folks accidently activate Siri. Or Google. Usually to laughs. If someone is wanting to be nefarious... I'd be more worried about the people you have over to your house, than the people's houses you go to.