A key concern I have (though it doesn't stop me blocking ads) is that I won't know if a site is normally full of the worst sort of ads (malware install attempts, auto-playing video & audio, tracking up the wazoo & back, ...) and I could send a link to people who are going to be affected because they are not protected by similar blocking.

So not a downside directly, but a risk of lacking awareness.