You can just mute the connected speakers and it's the same as having Siri off. Or get one of the ones that requires you to push a button (the Amazon Tap).