News for hackers. Stuff that matters.
Re:Why does Gmail hate my domain? by ceejayoz on Wednesday July 08, 2015 @06:43PM (#9855102)
If the domain is bitbin.de and the mail server is hosted on the same server, it's probably Hetzner being scorched-earth for deliverability, just like any other major hosting provider will be. Email from AWS, Rackspace, Heroku etc. get a drastically higher starting spam score because people fire up servers there to spam.

Running the email through something like Amazon SES or Mandrill would probably be helpful. Both have generous free tiers.

Re:Why does Gmail hate my domain? by jsnell on Wednesday July 08, 2015 @06:59PM (#9855167)
I run a server on Hetzner that sends maybe 100k messages a week, and appears to have no problems with delivery. It doesn't even have SPF or DKIM set up. And I'm pretty sure that if messages were not getting delivered, users would complain. At least that's exactly what happened the one time there was a configuration screwup [1] that made some email providers reject mail from that server.

So I don't know if that theory explains the issue either, though it could be a component.

[1] https://www.snellman.net/blog/archive/2014-12-05-how-buying-...

Re:Why does Gmail hate my domain? by ceejayoz on Wednesday July 08, 2015 @08:18PM (#9855473)
> I run a server on Hetzner that sends maybe 100k messages a week, and appears to have no problems with delivery.

That volume (assuming non-spammy content, of course) would over time build up a positive reputation for your IPs, outbound addresses, and content.

A new or low-volume IP on Hetzner is going to have a harder time overcoming the initial suspicion Gmail saddles it with.

Re:Why does Gmail hate my domain? by massaman_yams on Wednesday July 08, 2015 @08:57PM (#9855630)
That's probably more true in general than it is in this particular case. Gmail does IP reputation differently than other major ISPs, and domain/content reputation are very different beasts entirely.

Gmail's approach with a low-reputation IP is generally to throttle it aggressively; mail still gets through, just not very quickly if you're sending in bulk.

Domain reputation is not initially distrusted unless it's a brand-new domain (registered in the last 30 days or so), and generally doesn't come into play unless there's very unusually poor metrics (think sky-high spam complaints) associated with the domain. Content reputation is applied in a similar manner.

The headers I'm looking at should most likely give Gmail enough distinct identifiers to work with to isolate it from broader reputation issues at Hetzner, but the post only included partial headers, so I can't quite say that conclusively.

Re:Why does Gmail hate my domain? by fuzzix on Thursday July 09, 2015 @04:08AM (#9856698)
Always worth checking if the IP you inherit is in some DNSBL. A number of Hetzner IPs I've gotten were tainted.

It's oh so much fun trying to get them removed.

Re:Why does Gmail hate my domain? by seiji on Wednesday July 08, 2015 @07:56PM (#9855405)
Amazon SES or Mandrill would probably be helpful. Both have generous free tiers.

But you are still sending all your outgoing email through an external 3rd party. Not ideal. Also, these days every email is a top-reply orgy of all correspondence ever sent, so your SMTP relay essentially gets a copy of your entire conversational inbox (via "QUOTE ALL THE THINGS" reply garbage) as well as your outbox.

Re:Why does Gmail hate my domain? by drzaiusapelord on Thursday July 09, 2015 @10:32AM (#9858019)
While I don't doubt this, I often see from DIY types some really lazy smtp habits. If they don't have their reverse DNS matching the forward DNS as well as the HELO name matching that as well, then, yes, that will be a major red flag for anti-spam systems. Reverse DNS is pretty important and weighed heavily.

smtp delivery is something of a black art, but at the very least those things need to be set correctly. Its also worth mentioning that if you have 'spammy' applications that opt people in to mailings then a lot of people just get pissed off and click the 'junk' button over and over instead of unsubscribing properly. So think twice before auto-subing customers or enabling 'mail me replies' by default on forums and such.

I also learned the hard way that once you're on Cisco Ironport's spam list, then you really can't get off. There's no one to talk to. Its supposed to be automated. In practice, that means months before your domain is unblocked. I find most Ironport admins just use that sole list even while Cisco recommends they use many and weigh the average. After a stupid config change that opened our smtp to relaying, we got off everyone's blacklist but IronPort. IronPort is pretty much the defacto standard in several types of enterprise. My fix? Get a different IP address for the mail server.

Re:Why does Gmail hate my domain? by cmwelsh on Wednesday July 08, 2015 @06:50PM (#9855131)
Amazon SES has a pretty poor set of IPs due to the issue you mentioned about firing up servers to send spam.

Mandrill is top notch, just like other professional email providers such as Mailgun and Sendgrid.

Re:Why does Gmail hate my domain? by ceejayoz on Wednesday July 08, 2015 @08:43PM (#9855576)
Are you thinking of EC2?

SES (as with Mandrill, Sendgrid, etc.) runs on a separate block of IPs and is very strict about spam/bounce reports. Very easy to get yourself throttled or cut off entirely.

Re:Why does Gmail hate my domain? by quadrature on Wednesday July 08, 2015 @06:49PM (#9855122)
This seems to also be the case with DO, I setup a mail server on an instance there and can't escape people's spam boxes.
Re:Why does Gmail hate my domain? by brobinson on Wednesday July 08, 2015 @06:54PM (#9855145)
Check the IP against the spam blacklists: http://mxtoolbox.com/blacklists.aspx

You might have to spin up a bunch of servers to find one which doesn't have a blacklisted IP. Then you have to build the IP's "reputation" with the various providers before you can send a significant amount of email through it.

Re:Why does Gmail hate my domain? by massaman_yams on Wednesday July 08, 2015 @08:34PM (#9855530)
These days, there are only a handful of IP-based blacklists where a listing result in more than low single-digit impact to your delivery rate; moreover, among industry insiders, Gmail isn't known to use any public IP-based blacklist for deliverability.
Re:Why does Gmail hate my domain? by danielweber on Thursday July 09, 2015 @09:21AM (#9857658)
GMail may not use public blacklists, but they definitely have IP-based reputations that feed into other blacklists.

Just a few years ago, I personally experienced this sending email between two GMail accounts, and then I noticed that my (workplace!!) IP address ended up on a public blacklist within an hour. Fortunately I noticed fast enough to get it removed before any mail delivery was affected.

Re:Why does Gmail hate my domain? by brobinson on Wednesday July 08, 2015 @10:59PM (#9855954)
>Gmail isn't known to use any public IP-based blacklist for deliverability

Good to know! I bet they have their own internal ranking system which is far more accurate than the public blacklists.

Re:Why does Gmail hate my domain? by stonogo on Thursday July 09, 2015 @12:20AM (#9856123)
I'm curious to know why you think they'd be any more accurate than public blacklists. About a fifth of the spam that hits my domains originates from gmail servers, and the majority of people I know complain (when asked) about gmail's tendency to spam-bin legit mail.
Re:Why does Gmail hate my domain? by brobinson on Thursday July 09, 2015 @12:40AM (#9856175)
I would wager that Google handles a large enough cross section of all email traffic such that their internal statistics on the trustworthiness of domains/ips would rival any other blacklist system.

I've never had issues inboxing on gmail accounts using SPF/DKIM/DMARC + Sendgrid, even when sending 125,000+ emails (legit!) per day.

Re:Why does Gmail hate my domain? by massaman_yams on Thursday July 09, 2015 @02:51AM (#9856502)
Accuracy in spam filters is a difficult thing to measure, particularly if you don't have access to internal metrics on filter performance.

The primary limiting factor for most blacklists is not scale, but simply the fact that most of them have no more than a few data sources - most commonly spamtrap data. It's useful, but it's not a comprehensive enough data point to accurately evaluate mail.

Having dozens or hundreds of data points available - things like how many recipients open a message and spend time reading it, how quickly they seek out a message when initially opening their inbox, or a sending domain's pagerank - gives Google a considerable edge in assessing overall mail quality.

(Caveat: outbound filtering is often more difficult than inbound filtering - perhaps in part because there are fewer data points available when assessing outbound mail.)

Re:Why does Gmail hate my domain? by brobinson on Thursday July 09, 2015 @08:19AM (#9857400)
Agreed on all points!
Re:Why does Gmail hate my domain? by reitanqild on Thursday July 09, 2015 @02:06AM (#9856384)
> majority of people I know complain (when asked) about gmail's tendency to spam-bin legit mail.

I observed a former boss, who was very technically competent otherwise, using "mark as spam" instead of delete.

I guess if he could underestimate that button so can a million other people.

Re:Why does Gmail hate my domain? by quadrature on Wednesday July 08, 2015 @07:46PM (#9855365)
Thanks, I just assumed their whole block would be blacklisted. I'll try some more.
Re:Why does Gmail hate my domain? by ceejayoz on Wednesday July 08, 2015 @08:19PM (#9855478)
Not blacklisted, just treated with higher suspicion by default.
Re:Why does Gmail hate my domain? by arrty88 on Wednesday July 08, 2015 @06:51PM (#9855135)
FWIW, I am able to send mail effectively from my Linode box, (passing SPF and DKIM).
Re:Why does Gmail hate my domain? by kijin on Wednesday July 08, 2015 @08:52PM (#9855616)
Linode has always been very strict about keeping their IP ranges clean.

The fact that Linode's lowest tier has been $20/mo until recently ($10/mom nowadays) also helps. That makes Linode unnecessarily expensive for people who just want to burn IPs, since they can easily get an IP for $5/mo with DO or even less with low-end VPS providers.

Re:Why does Gmail hate my domain? by mattkrea on Wednesday July 08, 2015 @08:42PM (#9855569)
I ran my personal mail server on DO for about a year or so without issue. I would definitely try a new instance--I didn't even know this was a problem.