I am only mentioning bandwidth because someone could easily saturate my internet connections by extensively brute forcing, and that's another mild concern (perhaps those are things that kill my modems?).
In practice, biggest issue is that Gmail in particular will never rank your server as trusted because you are sending too low a volume (how's that for an anti spam measure?), and 1/3rd of recepients report emails ending up in spam folders. Curiously, only Gmail does that.
That's super easy to avoid. Use strong passwords.
Heat death of the universe will arrive before anything brute forces a 30 character password generated off /dev/random
I don't have any of my users (just family) set their email passwords, those are generated to be strong. These are not passwords that need to be seen or remembered by anyone, they just go into the IMAP client config.
I mostly run the mail for my family and myself, so the attack surface is reasonably small.
My network mostly only dies when one of the ISP-provided modems slows down too much without actually killing off the actual connection (so pings seem to keep working from my OpenWRT router to detect dead connections), which requires resetting that modem (happens only every few months, rare enough for me to bother debugging it: it's usually my dad who pings me early the next morning that his email is not working :)).