Everyone says this about self hosting email but I've had fewer problems self hosting email than any other service - though I don't use residential IPs I use dedicated servers or VPS. I've also seen plenty of comments from others who selfhost email whose experience matches mine.

I also see a lot of comments from those who have admittedly never tried, telling me that I'll be blacklisted and not even know.

I don't know if this is some kind of confirmation bias, or if there's just a very vocal bubble of people without experience talking about how difficult it is.