If you like to learn a lot about the internals of SMTP and mailservers in general and need/want a super reliable, fast and (if the setup is carefully crafted) secure mailserver - I like qmail.

My setup is qmail (SMTP) + vmailmgr + Dovecot (IMAP) and it was pretty easy to set up.

To start with qmail, I would recommend http://www.lifewithqmail.org/