I self hosted because I wanted to prefer not to be part of the huge o365/Gmail/iCloud monocultures.

Last year I moved my mail to an old fashioned shared webhosting account at Hetzner. Very happy with it!

How exactly is that solving the problem? If anyone does something remotely spammy from that ip, your mails are spam again too. And you probably got lucky that the ip you're sitting on was warm and trusted to begin with. You didn't really find a solution, the problem simply hasn't occurred yet for you or you are not aware of it yet.

GP is right. Self hosting email sending (and by that I mean any solution where you control the mail server) doesn't work unless you accept that you will randomly end up in spam folders and sometimes not delivered at all.

So either get a package at some of those smaller providers with a dedicated IP, or go to one of the big providers (Google, Microsoft) and use their highly trusted IPs, that everybody whitelists by default.

Instead, the hosting provider will operate a mail service for you, much like Microsoft 365, G Suite and the like. They will also take care of the IP reputation, SPF, DKIM and all.

However, I dunno why people only mention the cloud providers, while services like MailChimp, Sendgrid and others offer this for mass emails and companies like fastmail for personal users. The cloud isn't really necessary at all.

Many small busineses have their 'online' presence like that. If I have mail deliverability issues, it's not my problem any more, I can call Hetzner. And before I started using it I was sceptical because if Hetzner did not fix the reputibility of their IPs or the spam from their other customers, it would be a problem for me. But so far, it worked quite well!

The _real_ problem is reliably getting your 100% legit mail into your consenting recipients' inboxes.

It's amazing that having someone in your address book isn't enough in many cases. Like, why?

One of my clients is a construction firm specializing with churches. It is not infrequent for them to be communicating about a project with a church - often to a role account (e.g. [email protected]), which is step 1 towards being filed as spam - where the role account is shared with a dozen or more people. The building manager will check the email on Mon, Wed and Fri and every other day there will be a number of well-intentioned volunteers at least one of which will click on the spam button dooming my client's email into the abyss. Weeks later we find the church has gone with a different contractor, as they "could never get a response" from my client.

We will go so far as to donate a new IT/Cloud system to the church (small $$ compared with the project) just to ensure reliable communication. But then they think we are just trying to buy them.

Now my client has a blacklisted domain/mail server IP, and bids they send out are rejected as spam by the providers to other churches.

Makes no difference if we are hosting the mail server or if it is a 3rd party mail service. As soon as a customer is sold, we try and move them off email into a web-application framework for ongoing legitimate communication. Again a lot of resistance.

Also I'd expect all the firms competitors to be using Workspace or 365, so if using them gives no apparent protection, presumably they should be suffering from this as well?

The current situation is that we have technical solutions for authenticating smtp sending domains. But there will always be someone who flags an email too quickly or just wants to spite you. And so we’re back at square one.

There's almost no cost to send an email except time so even when n is large, this does not prove nearly as intractable as one wants it to be.

If they pass their log through a scanner, come up with a few dozen small servers that look fishy (e.g your IP is contiguous to some other spammer’s IP), and yours is rope in as false positive, you’re banned.

And from there you’ll have to convince Google that you’ve done nothing wrong.

Imagine the conversation with their support when you ask why [email protected]'s email was put in spam folder of [email protected]'s email, as both are hosted by Gsuite and presumably entirely within the Google network.

The answer, after two levels of support, was, "We don't understand why, but we can open a ticket with the developers." In the business world, you don't have time to wait for that, especially when non-technical business users are making mistakes or losing business because they can't function properly.

You would really think that Google would whitelist emails within the same custom domain which is paying for Gsuite service. Maybe that has changed, but it was definitely a problem 5 years ago.

And we're not talking about an email which had some copy/paste of spam, we're talking about a one or two line sentence giving an instruction or asking a question to a colleague.

> Bypass spam filters for internal senders

Is this what you are looking for?

The issue here is that developers and IT folk seem to think that email is easy because the know IT. Email is a complex, and old protocol that has many nuances and, believe it or not, phishers are smart; end users are naive, and often reckless. Spammers, be they benign or otherwise, can be incredibly lazy too, but don't think for one second that you, as a postmaster, are cleverer than they are. Spam and phishing are getting harder to beat. Businesses like Proofpoint, Cisco and Mimecast, along with the likes of Google and Microsoft, are investing heavily in trying to beat these guys, but the reality is that they are always at least two steps ahead.

Drives me nuts having to leave a UI to google (ironically) the issue to find a doc to enable a check box.

They still didn't care at all!

Just let ChatGPT send emails to itself, using mail server and client software that it wrote itself, and the rest of us don't have any need to handle email ever again.

Ingress - out of the box rspamd is pretty decent and it is rather configurable.

Egress - DNS (A,AAAA, PTR), (E)HELO, SPF, DKIM, DMARC. "IP Reputation".

There are absolutely no shortcuts and yet most of the problems I diagnose regarding email delivery will find a missing PTR record or a miss-configured (or non configured) HELO. You cannot be lazy when it comes to email. SPF + A + AAAA + PTR and (E)HELO is a minimum.

There is one thing that you cannot generally, personally mitigate and that is being on a deny list due to your IP address. This one is a bit more tricky to deal with. You might have to use a relay. Another mitigation might involve IPv6.

Any insight into why this is a giant ball of complexity? I’ve had to setup spf, dkim, dmarc a couple times now and man… design by committee?

The standards are fine in terms of achieving their goal, the hard part is getting the ecosystem to follow and make everyone's life easier.

Am I missing something not knowing what HELO is ??

AFAIK plenty ofservers will outright reject all mail if you only have an IP address and not a domain name there.

Extremely filtered emails is practically the same as a social network determining what I can see at this point.

Web standards have advanced to a level that walled gardens are replacing email. Email itself is still stuck on IE6 level standards. And SPF, DMARC and DKIM are a confusing mess to deal with.

Sounds like a salesmen who doesn't understand DMARC himself was trying to sell it by labeling it too complex.

From a purely technological point of view DMARC isn't that complicated. It just specifies how to treat DKIM and SPF results (with a bit of rather simple configuration). SPF is basically a list of ip's you own published on your domain (if the email was sent by that ip, you vouch that it was sent by you) and DKIM signs the email with a private key and you publish your public key on your domain so everyone can verify that this email was signed by the domain owner. SPF might fail if your mail gets proxied (as now its a different sender ip that you didn't vouch for) and DKIM might fail if the mail got modified including headers (because the signature can only be verified for exactly the original headers+content). So if you're sending email for someone else it gets a bit tricky, but for your own emails it's certainly not "just too complex" and boils down to a few line long configuration file, a list of ip's and a private/public key pair for signing emails.

But an admin UI is more than just for creating mailboxes. It can also manage quotas, the spam filter, provide setup instructions for the end users, …

Hey, we are starting this charity and need email for 15 people, what should we do? - order domain, create admin-account in the web interface, pass it on, done.

They could also provide Cloudflare-like spam and DDoS protection for receiving mail

Spam is a solved problem, thanks to SPF and DKIM. But despite doing all the right things, Microsoft and Google continuously block and rate-limit delivery.

Case in point: we deliver 20,000 booking confirmation emails every day, all requested by users and not spam. We have perfect Postmaster Tools metrics: absolutely zero reported spam, 100% IP reputation, high domain reputation, zero feedback loop spam, 100% encryption. But Gmail still rate-limits us daily, so their customers get their booking confirmations several hours late.

It's getting to the point where running an independent mail server is impossible. Yet another part of the open internet bites the dust.

I always prefer to isolate those things from the main corporate email and reputation anyway.

You don't want your main domain getting hacked and affecting customer emails or vice versa.

I’ve heard “there’s a lot of work being done on…” literally thousands of times. Yet outside of the crypto-sphere I’ve only met a handful of people who’s life would be impacted in the slightest if the entire thing disappeared.

Ethereum is eight years old and debates about use cases, etc aside I think even someone towards the middle of the debate can readily acknowledge progress, utility, value, and adoption has been incredibly slow.

Consider ENS - the last time I registered with ENS the process was just as confusing and convoluted as ever, only in this instance with gas fees I paid around $300 for the name. Oh, and the various transactions kept failing - leaving me wondering (per usual with crypto) if my funds went up in smoke, off to some scammer, etc.

Like so many times in crypto before I just assumed the funds were gone but like I said I DO follow this space and DO find it interesting so I consider burning crypto left and right a relatively cheap education/hobby. Plus I get to tell cool stories like this one on HN ;).

I can’t imagine how creaky, bug ridden, impossible to scale, nearly impossible for mortals to use, and inevitably completely overrun by scammers and frauds an Ethereum (or any crypto) based mail and messaging system would be. No. Just no.

With every day, week, month, and year that passes hearing “We’re building! People are working on XYZ!” when meanwhile the only thing people actually see is the latest headline for some crypto criminal, fraud, scam, collapse of last years “use case” (NFTs anyone?), etc.

I’m not saying it’s dead but I think it’s pretty safe to say the window for anything approaching mass adoption, credibility, and real utility has closed.

“Stop trying to make fetch happen! It’s not going to happen!”

MetaMask is by far the most popular wallet used for all things Ethereum and Web3. They touted "30 million" MAUs[0] during the peak (3/22) of the last crypto craze/bubble. Let's put that in perspective - there are an estimated 5.3 billion people on the internet. 30 million MAUs represents 0.06% of worldwide internet users. Facebook (which has been in rapid decline) has at least 2.6 billion MAUs. One social media platform vs an entire ecosystem (MetaMask can easily be characterized as the official onramp to Web3). Literally a half of a half of one percent of the TAM (internet users).

Want a (somehow) even bleaker picture? According to DappRadar the top 10 most popular Dapps[0] COMBINED have a lifetime total of 888k wallets (all numbers rounded up) that have ever interacted with the Dapp in any capacity. That's 0.00167% of worldwide internet users and I'm sure the MAUs for these Dapps are even more pathetic. Additionally, it's widely known that wallet != user (many users use multiple wallets and addresses). Looking at these Dapps none of them have any utility other than "DeFi" (swapping tokens around), play to earn games, and straight up gambling. To look at it another way, an absolute best case (impossible) scenario for MAUs of these apps worldwide is roughly the equivalent of the entire population of a small US city. Have you heard of Lee's Summit, Missouri (to pick one)? Yeah me neither and the same could be said for these Dapps.

All of the building in the crypto ecosystem is by crypto zealots for crypto zealots which the MetaMask and Dapp numbers show is an absurdly tiny (so small you can barely wrap your head around it) population. Crypto (other than gambling on exchanges) is an extremely obscure fringe religion.

[0] - https://decrypt.co/95039/metamask-consensys-30-million-users

[1] - https://dappradar.com/rankings

Those days are long gone, and the reason why is that monopolistic companies hate open protocols like email and want to 'monetize' everything they can.

Sure, you can buy domains but that leaves some paper trail and generally you will see added spam scores for new domains.

So SPF and DKIM don't improve spam scores on their own, but the give a much more reliable path to building reputation.

After the great Gmail exodus two years ago (when they killed then unkilled legacy domain accounts), I moved to MXRoute — and the amount of unfiltered spam I get is insane. It's a daily nuisance. I have added about 200 filter words now, blocked hundreds of e-mail addresses, but it's next to impossible to filter out sometimes.

From conversations and threads online by the MX Route admin, it seems like this is a nearly unsolvable problem, still. Spam evolves so fast and uses throwaway one-time approaches only, so once you've filtered for a pattern, the next campaign looks completely different.

So, yeah, sorry about the long response, but I don't think Spam has been solved, at all, at least not if you're using an e-mail that's been around for longer than a few years (and appeared at least in one leak).

What's even crazier is that they are so confident in their spam filter that they simply reject mails classified as spam. But that hasn't been a problem since the few years I have been using them as well.

The only filter I have is a couple of regular expressions to block the most lazy of spam at submission time. Even without that I got less than 50 spam mails a day, usually quite a bit less - and almost all of those are automatically sorted into my spam folder. How long does it take you to glance over a low two digit number of emails each day? Not a lot for me.

I think the "spam" problem is overrated. Do you get more spam than me or do you just have different expectations?

Spam is only “solved” on big providers because they mostly accept mails only from other big providers.

---

I would argue that “running an independent mail server” and mass-mailing are two entirely separate concerns.

> Spam is only “solved” on big providers because they mostly accept mails only from other big providers.

I would say that only accepting emails form other big providers isn't a good solution then.

I've reasons to believe that Gmail users get more spam emails as they report these.

It would be nice if such outsourcing went into other parts of the net (e.g. captcha).

There are approaches that work well, on a small scale, but "solved" is a step too far.

This is an honest question, in my opinion the only way to make Google more friendly with self-hosted mail servers is their users complaining about or leaving Gmail.

I'm self hosting email servers for multiple domains for now nearly 25 years. When moving to a new hoster with new IP addresses (especially Hetzner, as it is cheaper and seems to attract more malicious people), I have to contact Google, Microsoft, Yahoo and one or two other big providers to clear my IP address range. This has to be done only once and usually is resolved within hours (faster for automated sites). The bigger ones have a web page to do this, others can be reached via email and are always replying quite quickly. So after a day, everything works and once it works, this never changes and mail gets accepted.

These servers are being used by multiple users and businesses and none have reported any problems. For decades.

I am only using dedicated servers though and using everything in the book: DNSSEC, DMARC, SPF, DKIM, proper DNS records, TLS with valid certs and a correctly behaving SMTP server...

On the other hand, I'm also sending email for a client through AWS SES, and those emails are regularly blocked due to their IPs being on a blocklist. And indeed I also get spam delivered through AWS SES. I understand that it is a cat & mouse game for mail service providers to prevent spam being sent through their services. My point: switching to cloud providers doesn't magically solve your mail problems.

Aonther issue with large mail account providers (gmail at least), is that they will accept & deliver your message, but just "deliver" it to the spam folder. It's so annoying if mail is accepted but you don't know if it is actually seen by the recipient. This erodes trust in email. I understand mail servers don't want to tell spammers whether their email is accepted/rejected, but I would prefer a few more spams in my inbox if I at least won't miss emails.

that's what those embedded 1x1px images are for

There is also no guarantee that the image being loaded means that the user opened the mail and it wasn't just preemptively cached by the server/client. There is also no guarantee that the user opening the mail means they read it.

I'm not here to guarantee anything, just mentioning the technology exists and is still widely employed. there are no guarantees with messaging app ticks either

I was in a similar boat, though I bailed much longer ago than you. I think "Big Email" is happy to let enough spam flourish in the world so that it presents enough of an excuse to tighten the screws to make it effectively impossible to run your own server... just so that more and more mail goes through their servers to be mined for personal data. That's why I don't use Gmail any more, but it wouldn't shock me to find out that Apple is doing something with my email data too.