Setting up an SMTP daemon and IMAP/POP3 server is rather easy, the most difficult decision being which SMTPd to choose (I like Postfix, but Exim is Debian’s default and hence supposedly good as well, others like Courier) and then reading the relevant manpages :)
If you search for, e.g., "Postfix tutorial", you'll find plenty of reasonably good guides.
Then again, you have to worry about redudancy. How would you feel missing an important email, due to your VPS/Dedicated server being currently ddos'ed or unreachable for other reasons.
In addition, do not forget about backing up.
So, that again is in no way "easy 1 hour" task, especially for those who are doing it first time.
At which point all your data is up for grabs by your VPS provider?
I host my own mail (for values of "my own" where mail server is hosted on a vps on a foreign server), but I think it's a stretch to say it's trivial to do so.
I think one of the most serious problems the Internet is facing is the growing split between "servers and hosts" and "consumers and clients" -- and the asymmetric bandwidth (both adsl and cable). It favours designs that are centralized, not because of sound architectural reasons -- but because of inferior infrastructure.
On that note, is there a down-to-earth guide to setting up your own e-mail host?