For years I avoided to use any external service to decide whether its Spam or not. But about 2 years ago I started to rely on some of the external Blocklists.
Till today I have no problem sending Email. Even as I don't use DKIM or DMARC.
I use blocklists in my self-hosted setup but only for the purpose of adding header fields that my Bayesian anti-spam filter can use to classify messages. I don’t reject anything out-right, aside from attempted spoofs of the domains my server is authoritative for. Everything is received— it just may end up in an “Unsure” folder if it seems too shady for the filter to put in my inbox.
I get just about no spam at all (<10 per month, maybe).
That would be an interesting research project. The domains getting most spams on my servers are the ones that are old (20+ years) which I guess makes sense.
Checking the rspamd logs for the last month gives just shy of 8500 emails with what I'd consider "definitely a spam" score. There's probably another 1000-1500 sneaking under that.
> That would be an interesting research project.
Indeed. When you say get spam, do you mean pre or post-filtering?
Over the last 30 days (I don't keep them longer than that) I've received 43 spam emails which were sent to the spam folder (so I wouldn't ever see these, other than because I went and looked now for the sake of this discussion). In the same time period there was only 1 spam email which was missed by spamprobe and made it into my inbox spool.
There's a fair amount of would-be spam that gets blocked during the SMTP transaction due to things like bogus host in HELO, etc. I don't keep any stats on those, I did at one time but it was too much noise.
My email isn't secret, it should be on every spammer list I'd guess. It has been the same since the mid 90s and is all over usenet, email list archives, websites, etc and I've never made any effort to mask it.
All my numbers are post-rspamd. There's still sieve after that which does the "is it scored more than X?" to redirect into spam/not-spam.
> There's a fair amount of would-be spam that gets blocked during the SMTP transaction
Yeah, the blocklists and protocol strictness rules definitely cut down a lot before it even gets to rspamd.
My mx hosted with hetzner also runs rspamd. Of the 32k mails received in the last month, 40% were rejected (postfix DISCARD, so the sender sees the mail as accepted but its sent to /dev/null - this only happens to mails scored very highly as spam, or sent to a spam trap address), 10% were greylisted and 1% were delivered to the spam folder.
So I'm also receiving 10s of spam per day, but they're all delivered to my spam folder with rare errors.
With the exception of an issue delivering to AT&T recently, I haven't had any outbound deliverability problems in a few years, but then again I don't send very much mail at all - perhaps I'd have more trouble if I did but most of my mail is incoming.
Mostly. There's probably 3-5 spams a day which get into my inbox; also 10-15 a day that wrongly end up in the "maybespam" folder (generally bulk that I'm not worried about seeing 100%.)
The old lists may be more plausible.
Some spammer once associated 'Robin Bennett' with my email addresses at some point 20 years ago and kept reselling it. Never used that name, didn't even know Bennett was a surname. I assume they linked me up with that name to fill up empty cells in their list and make it look more plausible.
It's a good way to filter spam. The last decade almost all spam calling me 'Bennett' is from some US political group, which mostly reminds me that it will be a long time before anything GDPR-like will pass in the US..
But almost all spam goes to that old 20 year old email address I don't use. My current email addresses are much cleaner even after 10+ year of use
Not quite sure I follow? Unless you mean "only allow emails to specific email addresses you've noted down", in which case, yeah, that works but also means a lot of admin when you want to use a new one (plus there's 10+ other people who use my servers for email, not just me.)
> unless you publish your address globally?
At least one of my email addresses has been published globally since ~1995. Others since ~2000.
How would I automate "[someone with an account] just entered abc-xyz@domain into a web form to subscribe to something, add that to the valid alias file"?
Not really - that's a limited use alias. Doesn't really work for when you want to keep getting email from places. Also if I'm reading that correctly, spammers can just send mail to <randomword>.20.<knownaccount>@spamgourmet.com and you can't protect against those 20 spams.
But I don't find that burdensome at all, in fact I'd prefer that gmail let more stuff through to my spam folder instead of swallowing things that it misclassifies, because I have seen several instances of lost legitimate email, something I still don't understand how is deemed acceptable to the people who wrote it.
But I need to vote with my feet instead of complaining.
That's funny since on gmail I get tons more spam than on any of my self-hosted domain addresses. What's worse, I see emails in gmail misfiled as spam when they are not, which is far worse. I don't ever see that happen in my self-hosted system.
Both result in other small mail providers getting blocked for arbitrary reasons with limited recourse.
You also either need to apply the greylisting to some larger IP range (rspamd e.g. apparently uses /19 by default for IPv4) or otherwise specially handle some of the bigger mail providers, because some of them rotate through their servers between retries, so you could be in for a quite a long wait if you do per-individual-IP greylisting.
The biggest culprit I noticed this with was Amazon SES – a former mail provider of mine used per-individual-IP, non-configurable greylisting, and any mail sent through Amazon (which isn't just Amazon itself – quite a few companies are using Amazon SES for transactional mail and suchlike) would consequently almost always arrive several hours late (however randomly long it would take Amazon to finally re-use an IP during a subsequent retry attempt).
Even more infuriating, my mail provider's support would then claim that it wasn't their fault and they didn't know anything about any supposed greylisting.
Mostly. But then you get the "click the link in the email within 10 minutes" problem. There's also a non-zero number of "our mail didn't get through first attempt, oh well, give up" people. From running GL on my servers over a couple of years, it mildly cut down spam (on top of blocklists and fail2ban) but I'm now wavering over whether it's worth the hassle.
Still, I've given up on it since plenty of email senders are not standard-abiding (they fail to retry), and I've kept losing email. I only caved in in the last 12 months after 15+ years of doing graylisting.
That is the reason why I switched on some external block lists into the mix.
Both rely on filtering out non compliant senders, but postscreen's filtering might be less disruptive. Are there spammers out there who cannot pass a graylist but can pass postscreen ?
But much more importantly, that question is orthogonal to my argument. Using blocklists is a good way to cut down on spam, the fact that it might block some trivial percentage of people who for ideological reasons might or might not be on those lists isn’t the receivers problem.
You want this to not be the case? Contribute meaningfully to solving the problem in a more effective way. Don’t blame the people who just don’t want the spam.
I pointed out that adopting blocklists just makes it harder to operate a mail server as a small provider, and nobody in this thread appears to disagree with that assessment. They instead seem to take issue with the tone of the message.
I don't think it's the tone. While the inability for senders to get off blocklists can be true, you're still not addressing why the cost to the sender to not be blocked should have higher priority than the cost to the receiver to avoid blocklists to make email admin more of a burden.
The gp you first replied to (PinguTS) is running a personal mailserver and resorted to using blocklists because reducing spam -- at the cost of some legit people not being able to send email to him -- is a tradeoff he's willing to make. You haven't convinced every user running mailservers that they should increase their spam burden because some small providers can't get off blocklists.
As another example in another communication channel... Here's a similar "blocklist" for cell phones that some users take advantage of: https://about.att.com/pages/cyberaware/ae/cp
Are "legitimate" phone numbers getting caught up in that block filter?!? Of course. But phone owners are trying to stop spam telemarketing calls. Telling them that some phone users are incorrectly on AT&T's list isn't going to convince cell users to quit using the block filter. They're willing to live with a few legit callers getting blocked.
What's the ideological reasons here?
I wouldn’t expect sending email to be the problem. But I’d be surprised if it’s delivered.
To those that still persist, there is a page I found recently that helps you make sure that your outgoing mail is configured correctly: https://appmaildev.com/en/dkim. They generate an e-mail address, you send a mail to them, they do the check and display results (not affiliated, just a happy user).
I will check out your recommendations. Thanks
Google? No problem. Comcast? No problem. Charter? No problem. AT&T? Problem.
gmail on the other hand does what others said, report smtp 250 and silently discard some emails. (mostly those that lack DKIM)
We also should keep mail for pseudonymous and anonymous user authentication. There are a lot of threats to the free internet right now and user account consolidation is one of it. I agree that people should keep hosting themselves. Someone blocked you? Their loss, let them rant to their internal IT about it. Corporations or institutions that use Google as a provider should be seen with scepticism.
To host anything beyond those protocols and/or on more powerful hardware is often counter productive.
The problem is getting the ports opened, you need to fight for that right even if it makes spam worse in the short term.
Fight for external IP, ports and static IP in that order.
Also does it help to use Google/Azure to host with regard to IP reputation with Gmail or Outlook?
v=spf1 mx -all
This tells me only to accept emails from 'MX' entries for that domain. So I query 'MX' against the DNS server and I get a list of A records, which I can get IPs from. If the IP is in the list, spf passes. Otherwise it fails, mark as spam.For DKIM, when the email was sent it was signed with a key by the sending server. It is identified by a UUID in the incoming email. So the receiving server again queries DNS for TXT <UUID>._domainkey.hnemail.example and receives the public key as a response. Signature verification passes? Accept email. It fails? Mark as spam.
This doesn't have a lot to do with IP reputation. This is different. If you are a very large email provider, you might develop custom spam filters. IPs are allocated to 'autonomous systems' i.e. who actually uses them and hands them out to users, and depending on the business you might make some decisions about reputation. For example, if the IP address is part of a consumer ISP block that is handed out to users of broadband, chances are high that if they're sending email, it is probably a Windows PC compromised by malware.
Similarly, you might decide some ASNs are better than others. Some hosters are more liberal in what they will accept, such as VPN endpoints, tor nodes and such and as a consequence of this more spam comes from these ranges.
Rightly or wrongly, larger email providers try to add these extra filters to the process to protect their users from spam. This obviously sucks if you are genuinely trying to run an email server on your symmetric home fibre connection with a dedicated IP, but that's the world we live in.
I can't make any general statement on which providers might be best, and some people will have no issue whereas others will find themselves unable to send anything. I don't work for Outlook/Microsoft or Google and never have, so I don't know exactly what rules they use, and in all likeliness they shift constantly depending on spammer patterns. I can only say I've found running from a small DC to work pretty well.
Few people think about key cycling for DKIM as it isn't a built-in requirement at all, so once a UUID is set they just keep it until some point in the future that may never happen when they need to revoke the key because the private half is compromised.
Agree with a sibling comment that many major providers fail to operate the SPF/DKIM/DMARC tools they insist you do.
Each to their own, but ultimately if we don't hold on to the freedom to operate our own mailservers, it will be taken away through inaction. This means doing some things right: DMARC, DKIM, SPF of course, server maintenance, good password policies and of course IP reputation. The best way I can recommend for IP reputation is to use a dedicated provider or VPS provider that disallows things like VPN endpoints, where it is less likely they'll assign an address with a poor reputation. A good provider might also ask you what you intend to host, and you might be able to discuss IP addresses with them.