For myself, I'm not too worried about privacy (if I want that I'll use PGP). I'm more concerned about vendor lock in. Because as long as people are sending you mail to a domain you own, you can change the sending email provider to any other one that allows a custom From: address. Now from a practical side, I currently let Google Apps completely handle my domain's email, but I can yank it off there at a moments notice if I had to (i.e., if Google decided that I'm not allowed to host my domain there anymore).