Transactional email services will usually give you a disposable API key that can only be used to send email. If the key is compromised, the worst that can happen is that your server will try to send spam and get blocked by the email service.

Many of the same services can also be configured to issue an HTTP request to a URL of your choice if an email bounces. You could write a WordPress plugin to handle that if you want. No need to keep POP/IMAP passwords in your database.

There is no need for your website to go anywhere near your regular, human-accessible POP/IMAP/SMTP mailbox.