Also I am using windows (smartermail) so even a cheap vps isn’t that cheap.
However, I don't run it from home over residential ISP connection.
I mostly run the mail for my family and myself, so the attack surface is reasonably small.
My network mostly only dies when one of the ISP-provided modems slows down too much without actually killing off the actual connection (so pings seem to keep working from my OpenWRT router to detect dead connections), which requires resetting that modem (happens only every few months, rare enough for me to bother debugging it: it's usually my dad who pings me early the next morning that his email is not working :)).
I am only mentioning bandwidth because someone could easily saturate my internet connections by extensively brute forcing, and that's another mild concern (perhaps those are things that kill my modems?).
In practice, biggest issue is that Gmail in particular will never rank your server as trusted because you are sending too low a volume (how's that for an anti spam measure?), and 1/3rd of recepients report emails ending up in spam folders. Curiously, only Gmail does that.
That's super easy to avoid. Use strong passwords.
Heat death of the universe will arrive before anything brute forces a 30 character password generated off /dev/random
I don't have any of my users (just family) set their email passwords, those are generated to be strong. These are not passwords that need to be seen or remembered by anyone, they just go into the IMAP client config.
99% of people don't use their home computer to write programs. Some people who write computer programs at home cause trouble. Should we therefore ban programming languages for individuals?
Of course, how well your support requests are handled is another matter.
A tunnel to a vm hosted on a cloud provider is far from "running SMTP servers on a residential IP address" that OP was referring to.
On the outbound side, many accepters of mail will deny mail from SMTP servers on a residential IP address, as provided by the Spamhaus Policy Block List (https://www.spamhaus.org/pbl/) or equivalent. Unfortunately, there's not much one can do to get off this list - ISPs self-publish their own residential IP space onto it. In order to self-host from home, business-class service or higher is in order. Or, again, use the public cloud, and hope your compute instance's IP isn't on some other block list.
On the inbound side, though, many ISPs block inbound port 25. Since that's the agreed-upon port for mail receipt, there's no real way around this. Several times while I self-hosted off residential, I was able to call Comcast support and ask for the ban to be lifted, but not before fighting through several layers of level one support asking me to configure my mail client differently. The real kicker, though, is any time my modem was rebooted (I owned my modem, too!), I had to call in and request it _again_. My (very small local) ISP in upstate New York also performed this blocking, but _would not_, in the face of any complaining, lift the block.