Actually, it's not really enough to simply admin one's own mail server. Virtually all email is sent plain text, so the owners of any of the hops the email goes through (on the TCP/IP level) on the way to the mail server could relatively easily read your email.

Of course, it's even easier to read someone else's email if you own/run their mail server. But the bar is not that much higher if you're and ISP, a colo, or any of the probably dozens of hops your email goes through before it gets delivered.

If you really want privacy, you should use encryption. The even more paranoid should use encryption and mixmaster remailers. And the ultimately paranoid should use encryption, steganography, and some carefully chosen channels for transmission.