It's possible to leave servers. I was also irritated by the exact behaviour you described, until I figured out how to leave servers. You can also mute all notifications from a server.