"mailbox can only be accessed by me over an SSH tunnel"

Thats basically fantasy territory for an average user. If you consider the real world" abundant phishing, etc. then you will realise that in practice there is very little daylight between them.