If you get the wrong phase, you can hear it - it's louder.

So, if it's quieter, you're ok.

If the device makes the outside world louder, then yes, you might want to turn it off. (This actually happens on my headphones when taking a call.)