Oddly enough, if I simply drop email addressed to my domain registration email [7] that is NOT from my registrar, then the amount of spam I receive would drop about 70%.
But frankly, with the greylisting, the spam that does get through is easy enough to deal with. And I'm the only person who uses my domain for email.
[1] I started in the web hosting business in '96 or '97 so this is pretty much old hat for me.
[2] Definition: http://projects.puremagic.com/greylisting/
[3] Software I use: http://www.x-grey.com/
[4] http://boston.conman.org/2015/04/12.1
[5] http://en.wikipedia.org/wiki/DNSBL
[6] http://boston.conman.org/2015/05/11.1
[7] Yeah, I don't bother with the privacy settings on the domain registration.
It's all very well saying that the amount of spam you get is manageable, but that doesn't really matter if your own outgoing email becomes unreliable because you aren't using a recognised provider.
It's unfortunately pretty hard to tell the difference between people being slow to reply to mails to gmail addresses, and google flagging your mail as spam for no discernibly good reason.
Anecdotally they don't care about anyone other than gmail users, so there's not much to do about it other than a) give up on the idea that gmail interops with standard email, b) hope that more people move away from gmail, or c) just tell people that you don't care that your email is filtered as spam because they're using an email provider (gmail) that doesn't deal well with email.
Over the years, there have been only two domains where my email had issues being delivered: AOL [1] and Google mail. For AOL, I had to inform AOL that I was responsible for the IP address of my server and promise not to send spam and that I take care of any spams from my IP address, etc. Annoying, but once done, no issues. Google on the other hand ...
I've had a few times where my email was being dropped by Google mail. Annoying, but the issues did seem to clear up on its own after time. I do have a gmail account [2] that I use for testing when I think there are issues but I don't recall the last time I had one. But I have had them.
[1] My dad is still on AOL.
[2] Google's spam catching ability isn't 100% either. I don't use gmail at all for email, so anything I get there is either 1) a mistake because someone is trying to email a person with my name at gmail or 2) spam. It's mostly spam [3]
[3] I just checked my gmail account. 140 new messages since I last cleaned it out in mid-October. All spam. Way to go, Google! (but to be fair, it shunted 726 messages to the spam folder).
1) have a valid PTR record for the email server.
2) Set up SPF records.
3) Just run email on it. If you do run a webserver on the same server, DON'T RUN PHP! In my experience [1] the single most abused attack vector on webservers is PHP and about 60% of the time it's abused to install spam sending software on the server and that leads to the IP address getting marked as a spammer.
[1] About fifteen years in web-hosting.
There's an odd email server here and there that suspects anything that isn't gmail, there was even a mail server once that required me to prove I exist, but those are very few, and normally single user installations.
The secret is that besides not sending spam, you must receive email. If you don't get enough email, other servers start suspecting you. And, of course, there are configuration problems - since I'm writing my own mail server, I get a few of them, but even on standard software you have to make sure you set it correctly.