I am only mentioning bandwidth because someone could easily saturate my internet connections by extensively brute forcing, and that's another mild concern (perhaps those are things that kill my modems?).
In practice, biggest issue is that Gmail in particular will never rank your server as trusted because you are sending too low a volume (how's that for an anti spam measure?), and 1/3rd of recepients report emails ending up in spam folders. Curiously, only Gmail does that.
That's super easy to avoid. Use strong passwords.
Heat death of the universe will arrive before anything brute forces a 30 character password generated off /dev/random
I don't have any of my users (just family) set their email passwords, those are generated to be strong. These are not passwords that need to be seen or remembered by anyone, they just go into the IMAP client config.