The frustration comes with scenarios such as the one outlined in this blog post, where small mail server operators get bullied even though they are doing everything right. I can completely understand not wanting to operate a mail server due to this situation, or not having interest in leveling-up server administration skills.
For those of us who do have expertise in running mail servers, it's a shame we have to deal with these obstacles.
To me, that's the biggest reason not to spend time running your own infrastructure for mail. Getting people to accept your mail reliably, is much harder than adjusting some postfix configuration.
The upsides, sharing many of the common features of all self-hosting, are things such as pure and total control, data privacy, ultimate flexibility of configuration (e.g., infinite and unfettered aliasing), customization of interface, high-performance, etc.