It's particularly disappointing because even just getting the process started was delayed when no one answered our initial inquiries of interest for nearly a month. That was when I first figured out it was because my e-mails were getting routed to the agencies' spam folders. At this point, I'm probably just going to have to give in and get a gmail address for the sole purpose of completing an adoption, and then stop using it and go back to my real address.
Who'd have thought this would be the thing preventing me from completely de-googling?
I've run into countless problems because of my messages being marked as spam. The worse is that now it as become socially acceptable to ghost people, I never know who ghosted me, the mail server or the person! I've run into "I ignore your email, get the hint" a few times when thinking I was marked as spam and tried to contact people through other means.
Right now, when you send an email, you don't know if it was received or not and if the lack of answer is a conscious decision or because of filtering.
Do you remember the bad old days of spam?
Left unchecked I think it might have destroyed the e-mail ecosystem entirely.
Regarding the block vs reject, that unfortunately gives a big tip to the people whose working hours are spent trying to defeat three blocks and send you more spam.
There is not reason at all to suspect that a message from a small server with proper SPF and DKIM that has never sent spam (for real or spoofed, see SPF), is spam. The other half is from small servers that sprung over night and have no reputation.
Half of the spam I receive is from google/outlook/amazon. They don't block each other, do they?
Thankfully all my customers have been super understanding and were kind to take the disputes back once they were able to locate the emails in the Spam folder. Also I finally decided to just use a "reputed" smtp server to fix my problems (feels like extortion but what can you do).
I've written this before, but it's so sad that something so important and vital like email is so broken and we are the mercy of corps like Google/Apple/Microsoft and everyone has their own secret policy.
DKIM, DMARC, and SPF do, though, and basically are table stakes if you want your mail (especially mailinglist messages) to go through to people at major providers.
Every post master knows that it is done. But it doesn't have to be that way, although it certainly can feel like it when a company like google decides to not budge on the matter.
Sure, if you want spam. Don't like it, get people to deploy DKIM, then the domains will be used for reputation purposes.
What the root comment says is BS, the industry uses these methods for a very real and practical reasons.
I would have thought that it's fairly self-explanatory that anti-spam measures utilize the strongest signals. If sender domain becomes that, it will get more weight.
So if in the future email providers could reject both SPF-less domains and DKIM-unsigned letters, IP's would definitely become less relevant. So, get people to deploy those things.
Even better - Google could help small mail server admins by actually providing the information that landed their e-mail in the spam folder. If the protection is tied to the domain, no spammer will be helped by this knowledge.
And I understand that maybe a new domain might be suspicious at first, but after a few years of unchanged ownership (backed by whois data) there is simply no reason to put any mail messages from these domain to spam. Whatever the IP is.
Stop making up excuses for them. They are negligent at best, malicious at worst. Can't wait till they get hit by a lawsuit over this.
They definitely take the IP less into account if other things are more trustworthy. Totally ignoring it would be short-sighted from them. There are many cases where the domain is fine but looking at the IP and its usage patterns helps prevent abuse. Be it misconfigured (and then abused) SPF, stolen DKIM keys, public website that's email-capable getting compromised, these things happen a lot.
> Even better - Google could help small mail server admins by actually providing the information that landed their e-mail in the spam folder. If the protection is tied to the domain, no spammer will be helped by this knowledge.
They have a significant amount of content-based filtering, knowing that helps spammer reword their crap and bypass those.
> I have proved that the e-mail is tied to my domain and they know that my domain is not spamming - what more do they want?
That's also part of the thing, you can't prove and enforce this for both envelope and header from at the same time. Not to mention how minuscule the amount of perfect and strict SPF+DKIM+DMARC is out in the wild. At this point in time IP's are a very strong signal.
> Stop making up excuses for them. They are negligent at best, malicious at worst.
I haven't made a single excuse, I'm explaining why things work the way they do. You calling it bullshit won't make it so.
If you filter by IP block (or address!), it might be a block that has changed hands and is no longer spammy. Or it might be a block from the Zen Policy Blocklist, which blocks ranges that the responsible ISP has submitted as domestic or retail blocks that are supposed to send outbound mail through the provider's smarthost.
If you filter by domain, that could be the envelope sender, the From:, the Reply-to:, or the domain of the SMTP client. Only the last is reliable; and you also have the IP address for the client. In my experience, the IP address is more useful, for longer, than the domain name. But any good blocklist should age quickly (i.e. old stuff should drop off the list).
Depends on your approach. If you hack IoT devices then you have a lot of IP's. If you hack Joomla sites, you have a bunch of domains.
> I don't know - is free domain tasting still a thing?
Yes. There are also discounts and stuff like that.
In your case, it's likely that your volume and sending patterns aren't consistent and trustworthy enough to keep track of your domain and IP reputation.
You have to understand that they get millions of letters from new domains each day, sent from compromised Wordpress blogs and the alike. If you want to be deliverable, you have to be consistent and not suspicious.
Or, more likely, there's some other mistake in your configuration somewhere.
The need to warm up new IP's has existed for a while and a lot of providers do it. Any postmaster with experience knows how and why it's done.
Alternatively - come up with something akin to D&B registration system so people can attest that they won't engage in spammy behavior.
Yes, and I'm saying what's the prequisite for that to happen. As long as it's okay (which it currently is) to send unsigned mail, IP addresses have larger weight. DKIM needs more deployment for that to change.
There's absolutely no way that IP-based reputation schemes will be deprecated before alternatives are viable. Sure it would be nice for a few people here, but no, won't happen before the ecosystem improves.
> Alternatively - come up with something akin to D&B registration system so people can attest that they won't engage in spammy behavior.
Already exists. That too gets abused.
If google is too aggressive about dropping mail and gmail users can't reliably get the email messages they want, those users should move to a platform that doesn't block those messages. Problem solved.
I've seen several articles in recent months about Google (and other providers) blocking too much, but they are nearly always about the senders and the burdens blocking and sender reputation places on them.
As much as I'd agree that senders have to do a lot to get mail accepted maybe this is working as intended more often than not and the fact that people aren't dropping their gmail accounts in droves seems to suggest that the people whose inboxes are directly impacted by Google's filtering are largely happy with how things are.
Most email is spam, and for the few senders who aren't just bitter that they can't force spam on the most popular email providers with impunity, I freely admit that this is an extremely frustrating experience for them and that it does help degrade the utility of email and contributes to the consolidation of useful email providers. The current situation isn't perfect.
The idea that we could send mail from any routable IP address, or send bulk email easily to anywhere, or even set up our own personal SMTP servers on a whim and still send email to any other network in the world with no deliverability issues is beautiful, but not practical.
Given the decades of scammers and advertisers abusing email, I'd much rather leave it up to the recipients to decide when a service has gone too far in blocking what ends up in their inboxes.
It’s really really trivial to automate buying a domain, get some IPs and setup DKIM, SPF and DMARC. How can a provider determine intent?
Gmails approach is to mostly use user signals, which in my opinion is the best way.
At a company I’ve worked for, they sent some mail using sendgrid to a poorly made mailing list and we’re in gmail spam for ages.
My personal ran email mostly stays in inbox.
I think it’s such a typical mindset that people with very limited domain knowledge think that people with a lot more domain knowledge are just being stupid.
The problem is not that the level of strictness that Google is applying is necessary to reduce spam; the problem is that Google doesn't care and is in fact incentivized to not accept email from non-Google domains.
Yes, and sender reputation is part of it :D that's why they can handle it.
> Also, Hashcash[1], which is a virtual silver bullet for email spam, especially when combined with existing trust mechanisms
Who do you think has to and can spend more compute or money, well-trusted Google or a small player? Far from a silver bullet.
> i.e. the less reputation you have, the harder the Hashcash challenge
Who do you think will have better reputation? I won't even start at how different people define spam. Though, I can promise you that on average SpamChimp would get to send a lot of spam for much less "HashCash" than a small player.
> is in fact incentivized to not accept email from non-Google domains.
Everyone is.
They aren't doing a very good job of using sender reputation. If you have a domain with a several year record of no spamming whatsoever, and with every outgoing message using DKIM, they will still start blocking you if other senders who just happen to have an IP address close to yours start spamming.
IP block reputation should only be used to set the default when dealing with new senders. If they have seen enough DKIM signed messages from a sender to know that the sender has a good reputation, IP block reputation should have weight 0 when receiving mail from that sender.
Google could do this without any increase whatsoever in the amount of spam that shows up in their customers' inboxes. All that would change is that their false positive rate would go down.
That's your limited perspecive, sorry.
> they will still start blocking you if other senders who just happen to have an IP address close to yours start spamming.
Absolutely, how would they know how that provider assigns those IP's? A lot of spammers use entire /24's.
> If they have seen enough DKIM signed messages from a sender to know that the sender has a good reputation
A lot of spammers have DKIM, it's not a good reason to allow mail from a suspicious subnet.
Pick a provider that deals with their spam complaints. That's the harsh truth.
You keep missing the point. Nobody is suggesting that mail be allowed merely because a domain is using DKIM.
What is being suggested is that if a domain has a proven history of not sending spam then blocking the domain's mail just because there are spammers in the same subnet results in blocking non-spam. Not blocking the domain does not increase the amount of spam that gets through because the domain is not sending spam.
Where DKIM comes in is that it allows receiving sites to distinguish between mail that is really from domain X and mail that is forged to appear to come from X. The latter can safely be automatically classified as spam. The former can be used to build an accurate history for the domain to determine if it sends spam or not.
> What is being suggested is that if a domain has a proven history of not sending spam
That is merely because of using DKIM! Everyone can create a 'history of not sending spam', including spammers.
With your solution someone can rent a subnet, warm up a bunch of domains, but then blast spam from the entire subnet without immidiately affecting the entire set of spam domains.
What you're proposing wouldn't allow trusting even signed mail from an already suspicious subnet. Be the domain with or without reputation, so are the spammers'.
> Not blocking the domain does not increase the amount of spam that gets through because the domain is not sending spam.
They don't know that before they deliver your mail to inboxes and someone does or doesn't mark it as spam.
i'm not even a big fan of hashcash, but i don't think you understand it, or how it could fit in as a component of a broader spam mitigation system. (for better or worse)
google wouldn't need to compute hashcash for their outgoing emails, because they have DKIM and a solid reputation. nor is its computational power somehow at odds with that of smaller players.
It's not that I don't understand, I know how it won't.
> google wouldn't need to compute hashcash for their outgoing emails, because they have DKIM and a solid reputation.
Great, so what's the point? Small players would have to pay Google to deliver mail :D
> computational power somehow at odds with that of smaller players.
As I said in my other comment, either botnet devices can send tens if not hundreds of spam letters, or it's going to be slow and/or expensive for all the legitimate small senders. You really can't have both.
You really don't understand it, because you're unable to provide counterarguments to specific explanations of how it would fit in with a larger anti-spam system, and because you say things that are clearly false to someone who does understand it.
> Great, so what's the point? Small players would have to pay Google to deliver mail :D
...such as this, which is false. Nobody's paying anybody anything - Hashcash is a computational proof-of-work challenge with zero money transferred.
If you're talking about a metaphorical "payment" of the smaller players having to do some kind of work - that's a feature, not a bug, because it allows the smaller players to invest effort into convincing Google that they're legitimate, and if coupled with a well-designed spam system that then takes that reputation and associates it with DKIM signatures/IP addresses, then the smaller players burn n cycles and then stop having to do so because they now have reputation with Google, while spammers burn n cycles continually for every message that they send because they keep getting flagged as spam.
> As I said in my other comment, either botnet devices can send tens if not hundreds of spam letters, or it's going to be slow and/or expensive for all the legitimate small senders. You really can't have both.
Yes, you really can, because you don't understand how anti-spam systems work. A system with Hashcash would start out assigning most IP addresses+hostnames/DKIM signatures with a reputation of 0, and in order to accept a message, would require the sender solve a Hashcash challenge inversely proportional to their reputation. If the message is marked as spam, the reputation takes a hit - if it's not, the reputation increases slightly (along with all of the other factors that an anti-spam system uses).
The legitimate servers quickly build up reputation and stop having to solve challenges, while the spammers pay the computational tax until the end of time, making it unprofitable for them to send most spam to most targets. The end.
I have provided specific cases where the previously proposed use-cases wouldn't work.
> they now have reputation with Google, while spammers burn n cycles continually for every message that they send because they keep getting flagged as spam.
> The legitimate servers quickly build up reputation and stop having to solve challenges
So the spammers send a few warmup mails. It's very naive of you to think spammers can't imitate legit sender behavior.
It's also clear you haven't seen or analyzed any significant amount of spam. The end.
It's clear to anyone who understands the basics of spam and Hashcash, and can use basic logic, that "So the spammers send a few warmup mails" literally doesn't change the effectiveness of Hashcash.
Let me state it again: every new sender has to go through a reputation ramp-up process, but spammers are uniquely penalized by Hashcash because they have to repeatedly because after they start sending spam mail, they lose their reputation and have to start that process again. Normal users do not have to do this - they start a new server, solve a bunch of challenges, and then they don't have to do that again because they don't regularly burn their acquired reputation.
You say
> It's also clear you haven't seen or analyzed any significant amount of spam. The end.
...but you're unable to use basic logic on publicly-known facts (the vast majority of spammers send large amounts of spam; the vast majority of users send small amounts of email; reputation is hard to gain and easy to lose) to infer the above or generate a logically cohesive counterargument. Every time you can't give an answer, you fallback to an argument to authority.
Sender reputation where you block smaller email providers indiscriminately without weighting DKIM signatures or identity (or just blocking based on IP) is not a way of "handling" it. Your comment is akin to someone saying "the government can handle poverty" and you saying "yes, by killing all of the poor people" - it's very clearly a non-solution.
> Who do you think has to and can spend more compute or money, well-trusted Google or a small player?
I don't think that you understand how Hashcash is supposed to work, or what it's supposed to solve. The computational cost is imposed on non-Google players - both smaller email servers without a reputation (until they build one up, obviously), and spammers. I mean, sure, if everyone implemented the protocol, then non-Gmail servers could technically request that Gmail do Hashcash challenges in order to receive email, but nobody's going to do that, because nobody believes that Google is going to send spam mail.
> Though, I can promise you that on average SpamChimp would get to send a lot of spam for much less "HashCash" than a small player.
No, you really can't promise me this, unless you're literally in control of Gmail servers and intentionally adjust the difficulty settings to fulfill this condition.
> > is in fact incentivized to not accept email from non-Google domains.
> Everyone is.
I don't see what your point is here. The fact that Google is incentivized to not accept email from non-Google domains is not a good thing - it's a bad thing. And, really, the above is false - everyone is incentivized to accept legitimate, non-spam email from all of their customers, which is not the same as what you said.
That's not true.
> because nobody believes that Google is going to send spam mail.
This sentence indicates you don't know much about who sends spam or not.
> The computational cost is imposed on non-Google players
You're going in circles. Do you not see how that especially is unfair to the smaller players? Spammers have just as much compute, it will just impose an additional burden on the legitimate senders.
So in your scenario a small startup has to wait tens of minutes (bunch of letters in the queue already) to send email confirmation letters. The spammer on the other hand does the same, it'll infect a few more devices, they've exeeded the startup's sending rate.
Alternatively, both the spammer and the hypothetical startup build reputation first. They'll warm up the domain, both get to send more in the same timeframe. So... we're at the beginning again, everyone warms up their domains and IP's like they do right now. PoW hasn't improved the sitation.
> I don't see what your point is here.
Because you haven't seen enough spam. On average everyone is incentivised not to accept mail from new or unknown domains.
> everyone is incentivized to accept legitimate, non-spam email from all of their customers
No shit, but also not the topic. Received mail is not perfect, that's what's the actual topic.
Still... it is a creative idea.
Either botnet devices can send tens if not hundreds of letters, or it's going to be slow and/or expensive for all the legitimate small senders. You really can't have both.
On the other hand, spammers are likely sending hundreds of emails per minute per server, so limiting them to 1 email per minute would have significant consequences for them.
I hope this demonstrates the disproportionate effect Hashcash (or modern equivalent) would have on spammers when compared to legitimate senders?
We use exactly the same idea for secure password storage. Password verification is slow, really slow, on purpose. This makes it "slow for legitimate password verifier" (the website you're logging into), but it also makes password cracking on a large scale completely infeasible.
Yes, in a scenario where you're a legitimate sender that sends very few letters compared to a bulk spammer. Apples to oranges.
You unfortunately ignored both the low-rate spammers and the high-rate legitimate senders. The former is barely hindered and the latter is PoW-walled into inconvinience.
If you now bring in reputation schemes, then those can be played by adversaries, you'll end up at step zero but you have made a large climate impact.
> but it also makes password cracking on a large scale completely infeasible.
But this is not the current scenario. Intentionally complex password hashing is not really PoW we're talking about.
Both of these seem like imaginary edge cases that don't really exist. I'm going to gloss over the oxymoron of low-rate spammers.
> If you now bring in reputation schemes, then those can be played by adversaries, you'll end up at step zero but you have made a large climate impact.
How do they do that? If we focus on the current state of things, high rate legitimate senders already have an established reputation so they don't need to change how they handle outgoing emails.
The only thing that would change is how large email providers such as Gmail handle incoming emails from small/unknown senders, by accepting a sufficiently complex PoW as an indication that the email is not spam. Over time this should increase the sender's reputation to a point where PoW is no longer required.
TLDR: Nothing would change for the big players, small senders are given an opportunity to prove they are legitimate, and spammers remain locked out.
I'm sorry? Both of those cases are very real and happen often, you don't get to just ignore them because of your lack of experience.
> How do they do that?
Many ways. Sending legitimate mail for example. You're acting like it's hard to show good behaviour for a while. It's one of the telltale ways I catch those fucks, they're doing it already.
But they also use their own inboxes to build up volume, hijack accounts, abuse SPF policies and DKIM misconfigurations. There probably are more ways to get a "trusted" domain I can't recall this instant.
Things regular senders can do, spammers can do, but they'll spend more effort to do it at scale!
You have demonstrated repeatedly how you have little knowledge of spammers' and others' behaviour, you handwave away all "edge cases" and say it will work. No, it won't.
All you'd add is wasted compute resource and time. Compute and time that especially spammers get for free by abusing others.
These patronizing "you obviously know less than I do" type comments that occasionally drop a few bits information aren't adding anything to the conversation.
If you haven't noticed, I heavily copy phrases from the people I reply to. If you don't like your tone being matched, unfortunate.
> Please elaborate on the intricate details and challenges of dealing with spam, from start to finish, as well as any proposed solutions to make self-hosting viable when dealing with providers such as Gmail.
I have, and you included have ignored it as "edge cases" and alike. What can I more explain when someone has taken the stance of being dismissive?
"Often" but as a tiny fraction of all sender cases, something that anyone who's actually worked with email spam would know, so you either dishonestly withheld that fact or you don't know what you're talking about.
Also, low-volume spammers are quite clearly an oxymoron - the definition of "spam" includes high volume. It's literally high-volume by definition: "unsolicited usually commercial messages (such as emails, text messages, or Internet postings) sent to a large number of recipients or posted in a large number of places"[1]
And, the vast majority of individuals do not send high volumes of email. This is a fact. It is an edge case.
> because of your lack of experience
You seem to be making a lot of arguments from authority, yet you don't even have the credentials to back them up.
> You're acting like it's hard to show good behaviour for a while.
They're definitely not. It's crystal clear that the usefulness of Hashcash is not dependent on it being hard to show good behavior for a while, and in fact its most effective in the long term.
> But they also use their own inboxes to build up volume
Something that Hashcash solves, because they'll have to spend a lot of cycles solving challenges to send non-spam email to build up reputation repeatedly, because for each "clean" IP/server they'll have to repeat the "reputation ramp-up" that every legitimate sender only has to do once.
> hijack accounts
Literally no anti-spam situation protects against this, so it's irrelevant, and I don't know why you're bringing it up.
> abuse SPF policies and DKIM misconfigurations
Then these need to be fixed. Google indiscriminately blocking smaller senders isn't a fix.
> Things regular senders can do, spammers can do, but they'll spend more effort to do it at scale!
Exactly - that's what Hashcash does, is force spammers to spend more effort on the things that regular people do, because spammers do it at scale far more often than regular people do, and so it introduces a disproportionate cost on them.
> All you'd add is wasted compute resource and time. Compute and time that especially spammers get for free by abusing others.
This conclusively illustrates that you have no idea what you're talking about. Spammers do not get resources for free - they have to either spend time to acquire botnets and hijacked domains themselves, or pay money for someone to do it for them. This is pretty clear to even people who aren't in the industry.
Spamming only happens if its profitable, and the reason why its still profitable is because the cost of sending a spam email is currently low enough. Hashcash directly increases the cost of spending spam email by making it compute-bound on top of needing to acquire domains, IP addresses, and machines, and spending time establishing a good sender record, while (if combined with a reputation system) barely penalizing normal users in the long run.
Let me state it again: every new sender has to go through a reputation ramp-up process, but spammers are uniquely penalized by Hashcash because they have to repeatedly because after they start sending spam mail, they lose their reputation and have to start that process again. Normal users do not have to do this - they start a new server, solve a bunch of challenges, and then they don't have to do that again because they don't regularly burn their acquired reputation.
The fact that you can't actually counter these arguments about Hashcash sounds almost like a spammer who's worried about the deployment of an effective system based on Hashcash.
Not really, no. I mentioned and you're dismissing high-rate legitimate senders and low-rate spammers. The former happens for example when companies send bills each month, they hit very high rates compared to their usual baseline. The latter is not the majority volume-wise, but is the most annoying - fairly logical that some spammers choose to fly under the radar and drop a letter or two in a minute. Do that during the night and you've landed a lot of letters in people's inboxes before you get reported.
> And, the vast majority of individuals do not send high volumes of email. This is a fact. It is an edge case.
It may be an edge case for you, but it must be accounted for. You can't wave it away because then people won't receive their mail. Just one minor example, some less tech-savy users emulate mailing lists with hundreds of people in CC. You can't say there that "oh this is my anti-spam solution but it doesn't account for you because you're an edge case"
> Also, low-volume spammers are quite clearly an oxymoron - the definition of "spam" includes high volume.
There's a difference between total volume and rate. You confusing the two is your problem. Not to mention that a dictionary definition is not the ground truth neither does it define "high volume". I'd also say unwanted unsolicited advertisement sent to 30 people is still spam, but you do you.
> You seem to be making a lot of arguments from authority, yet you don't even have the credentials to back them up.
From experience backed up with explanations, and you're dismissing them based on yours with no explanation.
> Literally no anti-spam situation protects against this, so it's irrelevant, and I don't know why you're bringing it up.
Incorrect. Looking at IP addresses suddenly behaving unusual is a great signal compared to the aggregate of a domain's total.
> Then these need to be fixed. Google indiscriminately blocking smaller senders isn't a fix.
They absolutely should be, but it's difficult and in the end you're not alone on the internet. Until people will, IP's continue to be used for spam classification.
And no, it's not indiscriminate.
> Exactly - that's what Hashcash does [...]
I think you misunderstood again and let me rephrase. Spammers spend more effort to lower effort spent per unit of spam. So in the end they spend less effort per letter than legitimate small senders.
> This conclusively illustrates that you have no idea what you're talking about. Spammers do not get resources for free
Your sentence conclusively illustrates you're clutching at straws. Declaring that something stolen hasn't been gotten for free is a shaky foundation to build your argument upon and pedantry at worst. The cost to acquire those resources is there anyways, some extra CPU they have to steal is not that significant.
> Hashcash directly increases the cost of spending spam email by making it compute-bound on top of needing to acquire domains, IP addresses, and machines. > Normal users don't have to do that again because they don't regularly burn their acquired reputation.
You're going in a loop again. We're comparing relative effort/monetary cost per email here. If one side has lowered the cost of effort by parallelizing things it's unfair to the side that hasn't. Consider the scenario where a new online store owner starts sending email or someone's script is compromising yet another poorly secured blog, IoT device or SMTP account.
Now a legitimate sender has built up all the reputation, after a few flags by recipients it was spent in an instant. Everything will become slow and expensive for the actual owner, once again. For spammers it was the tenth one that day.
Such a massive extra cost for the spammers. /s
That was just one example. Again it all boils down to the fact that regular users can't parallelize their actions in the same way spammers can - PoW would make it *relatively* more expensive (time+effort+money) to legitimate users per email.
> The fact that you can't actually counter these arguments about Hashcash sounds almost like a spammer who's worried about the deployment of an effective system based on Hashcash.
So many ad hominems, so little substance.
Disclaimers: I make sure to do everything else right. I have SPF and DKIM and my DMARC policy is to reject 100%. I also don't use IPv6 (DO Kube doesn't really support IPv6 well).
I have found other major providers to be much worse. Microsoft seems to rely almost entirely on IP reputation and marks everything as spam, even accounts that have marked messages as "not spam" many, many times. Apple outright blocks the IP range.
While on the other hand I fully agree with TFA: I have _never_ been able to send an email to Gmail from a IPv6 address and have it not end up as spam, not even to accounts where I already whitelisted previous attempts.
I don't think it's a reputation issue, since my IPv4 addresses likely have much worse reputation. It's as if they just handicap all IPv6 addresses.
I noticed that some providers completely block digital ocean IPs on their firewalls. So no way to get unblocked.
That's why your experience does not apply to the GP.
The real story is nobody cares.
Exactly. Why should the major players care about those too small to matter?
> Sure you would have to catch people buying a "good" domain that expired
Adding to my earlier point, Google and Microsoft are well-enough connected to the domain registrars to know when that scenario has happened as well. If they put any effort into it, they could reliably determine whether a new IP address for an established domain is legitimate or a fraud. But as we've said, why put any effort into it when the only people complaining are not important?
as far as how to solve this problem technically, I think a reputation system based not on domains or ips but on email certificates is the real answer here.
How would that help? Spammers can get certificates too. Maybe it cuts down on some of the misconfigured http email senders, maybe, but not enough to matter. Scam sites run https these days.
You can't use like age of activity of the cert to help because a) things get compromised, b) you need to rotate your certs frequently anyway.
It wouldn't help to fully trust, nothing would, it's a human problem, it would help to trust more.
Any wide spread certificate program will just have the email address as the identity, and it will be authorized by establishing control of the email (just like the majority of certificates used for https are domain control only, no organizational verification, not that organizational verification means much anyway). Anyway, identity is hard; there are many people with my name, including a Pulitzer winning author.
DKIM does not and DKIM keys should be rotated once in a while, but few do.
Not to mention all the websites that get hacked or the uber-cheap registrars.
At the point of receipt, when verifying via DKIM that foobar.com did indeed send this email, then update your spam statistics for foobar.com and you're good.
PS yahoo is the worst for assuming poor reputation from a sender. In my experience they just introduce massive delivery delays at the drop of a hat even when the sender has a stellar reputation.
You're right in one more way: both Google and Microsoft offer email hosting solutions for small businesses. Their main (and, if you've got more than 8-10 accounts or so, the only) selling point is that it makes managing email hassle-free. Making it as painful as possible for small businesses to host their email server helps these services tremendously. If they had real interoperability (either of their own accord or because it were forced upon them through regulatory measures), the biggest cash cows of these services -- companies that are well into "enough accounts that your own server would be much cheaper" territory but not large enough to afford or risk large infrastructure changes -- would evaporate pretty quickly.
But to reiterate my earlier point: Google and Microsoft have the computational and storage capacity to have a detailed history of all domains' authoritative mail servers behavior. They will know whether a domain has a history of patch negligence.
No, I think the far simpler explanation is that they just don't bother tracking reputation by domain. Or if they do, it's largely overshadowed by the weight given to IP-based reputation.
while I concur with all of the points you make, there is a logical and statistically accurate reason for some of these spam filters.
even if your no-open-relay, rdns, spf, dkim, dmarc, SSL/TLS and other configuration is absolutely impeccable on your smtpd, your only recourse in a situation like this is to change to a new ISP that does not have a poor IP space reputation in all the adjacent IPs.
That algorithm doesn't work. The internet is filled with parked domains that have "never been malicious". This just creates a new market for clean domains that you can use to evade protections. It makes spam a little more expensive, but it still gets through.
None of these tricks work. There are no tricks. All rules can be gamed. The only thing that can't be easily faked is reality: if Microsoft knows you're a big org with a well-managed IT group running your output email setup, then they know they can (probably) trust you not to spam their customers. If you are too small to prove that to MS in a scalable way, no amount of heuristic trickery is going to help you.
Google--a technology giant with algorithms and heuristics running most operations--is not up to the task of improving algorithms for email server reputation? No, they are definitely capable of improvement. It's simply not of interest to them.
I can hardly blame them because the cost-benefit analysis clearly says, "why bother?"
Of course anything can be gamed, but magnitude matters. I've had the same domain for 24 years, with many hundreds or thousands of email conversations between users of my server and those of the major email players over the years. I had to switch my mail server's IP address two years ago. Immediate zero reputation from many big players.
Incorrect, if they stopped spending all that money and effort to keep up, their users would get flooded.
> Of course anything can be gamed
Sending spam/marketing e-mails is a multi-million industry. Both on illegal and legal markets. It's a constant race.
This isn't the 1990's anymore, recipient domains aren't just homes to a few hundred undergraduates or a dozen programmers. Email hosts manage communication for hundreds of millions of customers.
How is it not easy to check a reputation database for domains when evaluating a mail server?
Using DKIM records, Google could cross-reference the reputation for the sending domain and, where applicable, recognize that the domain in question has never been malicious or negligent. And in that case, extend a probationary reputation sufficient to allow the IP to establish its own new reputation.
> How is it not easy to check a reputation database for domains when evaluating a mail server?
The point I'm making in this thread is that they have the ability to maintain a database of domains that have been proven to be trustworthy. I am not talking about a database of IP addresses, and I am not sure why that is being raised here.
Huh? With dataset of ~100k classified hams/spams I get like >99% precisison in identifying spam/ham with just bogofilter and 0 heuristics whatsoever (my mail server accepts everything, and I just use bogofilter client side).
I guratanee you MS has a dataset with > 100k emails, lol.
It's not like this is unsolved problem.
My god, are you joking or what? Spam, a solved problem? Far from it.
Google reads and indexes every email that they deliver to a gmail inbox. They probably have the worlds largest and highest paid staff of ML experts, and they resort to auto-flagging based on IP address?
Fix it.
Honestly, the best filter in my experience is Spamhaus Zen; it spots most spam with very few FPs. It's an IP blocklist.
In the end it's still a hard problem, so takes a bunch of work.
It's an unsolved problem for anyone.
> Why can I identify spam at a glance, with greater than 90% accuracy
I don't think you understand how terrible even 99.998% accuracy is at their scale.
> They probably have the worlds largest and highest paid staff of ML experts, and they resort to auto-flagging based on IP address?
You think they aren't using them for this? IP addresses very likely go into their model as well, when determining something is spam.
> Fix it.
They are and adversaries are breaking them again.
What does scale have to do with it, am I missing something? Wouldn’t a spam detection rate of 99.999% mean that only every 100000th spam mail would get through on an account basis, how’s that terrible?
The person I replied to said they can identify spam with 90% accuracy with a glance, my point was that it's abysmal as accuracy. Even orders of magnitude better it'd still be bad.
> Wouldn’t a spam detection rate of 99.999% mean that only every 100000th spam mail would get through on an account basis, how’s that terrible?
I didn't specify if it's per-account or a false positive rate, it's also actually not relevant to the point I'm trying to make.
When you receive billions of letters a month small errors start to matter and they're doing a better job than the person I replied to thinks they do.
From the perspective of an email receiver, I WANT things to be difficult for this group. Truly free email available to all would be unusable owing to spam. Oligopoly, I believe, results in the best experience for everyone.
Google and Microsoft have the computation and data storage resources to forever record how much spam they've received from the mail servers operated by any given small company or hobbyist. If a domain's authoritative mail servers have a history of never sending spam, when that company or hobbyist (for any reason) has to switch hosting providers and gets a new IP address, Google/Microsoft should recognize that the domain, and its referenced mail servers, has never been malicious.
But instead, the majors treat a new IP address for a long-established domain as entirely new to the internet and assign it zero, or even negative, reputation. And they don't care to address this because, well, why care about hobbyists and the rare small company insolent enough to try to self-host email?