You can make it completely impossible for a server (or people) to notify you in any way whatsoever through settings.

That's one of the positive features that keeps streamers and communities using it -- you can't bother people who don't want to be bothered.