News for hackers. Stuff that matters.
Re:After self-hosting my email for twenty-three years I have... by jwr on Sunday September 04, 2022 @04:21PM (#32717295)
I've been hosting my mail for 20+ years now, with minor issues. I guess I've been lucky.

Reading the comments here makes me incredibly sad. Every answer that tells me to use a provider misses the point. The Internet was created so that there could be many independent nodes, not so that everybody has to rely on one of several blessed providers. I should be able to run my own E-mail.

The real problem is lack of incentives. The big corps do not care about e-mail. It doesn't make money and isn't easily controllable. You can't turn it into a walled garden and lock users in. So, it gets minimal attention, and only defensive measures are developed.

Either we solve the spam problem, or things will get worse. The big tech corps won't solve it for us.

Re:After self-hosting my email for twenty-three years I have... by neop1x on Sunday September 04, 2022 @05:26PM (#32717860)
I have also been self-hosting email for 15 years and only had couple of problems at the beginning, mainly until my IP got enough reputation. I have been hosting it on a bare metal Supermicro server in a proper datacenter, though. It has reverse-DNS, SPF, DKIM, TLS, MTA-STS and even DANE with DNSSEC (on a self-hosted BIND but that's another story). It is implemented using Exim, Dovecot, SpamAssasin, DNSBL and Roundcube with OpenLDAP auth. I can recommend this awesome hand-on guide provided by Netherlands Domain Registration Foundation as a basis of a nice configuration https://www.sidn.nl/en/news-and-blogs/hands-on-implementing-...

I had some troubles with IMAP search. I set up CLucene, it was easy and enough for me (no need for Java Lucene). It just took me a long time to figure out why it wouldn't search a domain part of email addresses. It just required to set up the tokenizations in such a way to split words also on @ character, i.e. don't consider a full email address as a word. :P I also had some troubles with OpenLDAP until I finally decided to read the docs and examples there properly. Since then I have been using this setup happily and it appears I will continue to do so! I also share the LDAP with NextCloud btw.

Re:After self-hosting my email for twenty-three years I have... by jhugo on Sunday September 04, 2022 @09:35PM (#32719579)
> I have been hosting it on a bare metal Supermicro server in a proper datacenter, though.

This is a key difference. Many people who have a bad time self-hosting mail set up with a minimum-effort major provider like Hetzner or DO — you're guaranteed to have spammers as neighbours some of the time, and other networks will behave accordingly when deciding whether to accept your mail.

A real server in a proper datacentre is great — I do the same — but as a lower-cost option a virtual server with a small, clueful provider who knows their customers also works just as well.

Re:After self-hosting my email for twenty-three years I have... by RockingGoodNite on Sunday September 04, 2022 @11:36PM (#32720247)
I've run my email server in VM's at a variety of ISP's like GoDaddy, Linode and AWS Lightsail and never had an issue, well once when Microsoft banned a group of IP's in a drag net. I filed a request with them and they fixed it in a week or so.

Real hardware servers are not required to host email.

Re:After self-hosting my email for twenty-three years I have... by jhugo on Monday September 05, 2022 @09:57PM (#32731788)
> Real hardware servers are not required to host email.

I agree, and said that in my parent comment.

Re:After self-hosting my email for twenty-three years I have... by neop1x on Monday September 05, 2022 @03:32AM (#32721524)
I had tried it on a very inexpensive VPS before having my own physical server. The IP was blocked almost everywhere. Even though I filled out a lot of forms in Outlook, Yahoo, etc, it appeared that those major email providers had blacklisted the entire subnet of the VPS provider, and IP reputation seemed to have made little difference. Choosing the right provider is crucial!

A new VPS provider may work well and make you happy, but later on, many spammers may start abusing it to send spam. Then, major email providers begin to block the entire IP subnets that belong to your VPS provider and you are screwed.

Since spammers typically can't afford to buy and house servers, having your own physical or dedicated server will likely increase the likelihood of everything being fine for a long time or forever.

Re:After self-hosting my email for twenty-three years I have... by jhugo on Monday September 05, 2022 @10:02PM (#32731822)
Selecting the right host is very important. They should be unattractive as a host for spammers. Get this part right and you don't have to worry that tomorrow your neighbour is a spammer. The virtual/dedicated thing is not as important — many providers comingle virtual and dedicated servers in the same address space anyway.

One example of a good signal: if you cannot sign up online, but must make personal contact with the provider (via email or other means) to set up a server. It's a minor inconvenience for you — you're only going to do it once — but spammers expect to have to do it many times and won't bother.

Re:After self-hosting my email for twenty-three years I have... by reidrac on Monday September 05, 2022 @03:42AM (#32721569)
I don't agree, but it is true that some aggressive DNS RBL may include Digital Ocean, for example, just because [1].

The point is not that much that there are such blocking lists, the point is why admins use them. Surely that's blocking a good chunk of spam, but also legit sites.

1: "As you should know now: It is not you, it is your complete provider which got [redacted] listed."

Re:After self-hosting my email for twenty-three years I have... by jhugo on Monday September 05, 2022 @09:57PM (#32731785)
DO regularly shows up on RBLs, even "less aggressive" ones, because it's regularly a source of spam. Same with Hetzner.
Re:After self-hosting my email for twenty-three years I have... by jwr on Monday September 05, 2022 @10:31AM (#32724722)
I don't think it matters which provider you choose. Big corps ban entire IP blocks, and you will get spammers as neighbors both at a low-cost provider and at a "proper datacenter".

FWIW, I've been using Hetzner for the last 10 years or so (but bare metal server, not a VPS) and haven't had issues.

Re:After self-hosting my email for twenty-three years I have... by jhugo on Monday September 05, 2022 @09:56PM (#32731776)
It absolutely matters which provider you choose, assuming the provider is big enough to have their own address allocations (which is not that big, really — plenty of relatively small hosting providers all over the world have their own ASN and address allocations).

My recommendation is to use a provider that doesn't host spammers. Hetzner is not great at that in my experience, but sounds like they've been fine for you. Smaller providers who know their customers and are able to pay attention to how their network is being used are generally quite good at making things difficult enough for spammers that they go elsewhere.

Re:After self-hosting my email for twenty-three years I have... by GTP on Monday September 05, 2022 @06:39AM (#32722755)
How did you get reverse-DNS? When I tried self-hosting, that was a problem for me.
Re:After self-hosting my email for twenty-three years I have... by driverdan on Monday September 05, 2022 @09:52AM (#32724286)
If you're having trouble with a hosting company not setting it up for you name and shame. It's a basic service they should all support.
Re:After self-hosting my email for twenty-three years I have... by rootusrootus on Sunday September 04, 2022 @06:02PM (#32718142)
The problem is that collectively we love 'free' (at the point of sale) so much that we'll gladly allow gmail to just walk in and own almost the entirety of the email infrastructure. Then later we realize this gives them the ability to unilaterally make the rules, and we complain. But it's too late.
Re:After self-hosting my email for twenty-three years I have... by jhugo on Sunday September 04, 2022 @09:33PM (#32719558)
Also 20+ years, only very minor issues. The spam situation has gotten much better over that time, too. This topic comes up every so often on HN and I feel like the "never self-host, it's guaranteed to be a disaster, you have to use a centralised provider" crowd are just louder. Self-hosting my mail isn't something I think about much or talk about much. It's so obvious to me that it's worth doing, and it's extremely low effort.
Re:After self-hosting my email for twenty-three years I have... by HenriTEL on Monday September 05, 2022 @09:19AM (#32723967)
> It's so obvious to me that it's worth doing, and it's extremely low effort.

DMARC, IP reputation and spams are not obvious.

Re:After self-hosting my email for twenty-three years I have... by jhugo on Monday September 05, 2022 @09:58PM (#32731797)
Amazing — even though you quoted me, you still managed to reply to something I didn't say!
Re:After self-hosting my email for twenty-three years I have... by HenriTEL on Tuesday September 06, 2022 @05:17AM (#32734735)
You say that self-hosting a mail server is obvious and extremely low effort, which simply is not true. But ok, let's say that you happen to have a clean IP and no spam or dmarc issues. You still had to choose what to deploy, which is not obvious and actually setup/maintain your server which is not extremely low effort.
Re:After self-hosting my email for twenty-three years I have... by jhugo on Wednesday September 07, 2022 @07:26PM (#32758655)
Please read the comment you are replying to again. I said that it's obvious that it's worth doing, not that the work to set it up is "obvious".

Of course you need to learn some things, but it's not that much work. If you're the type that enjoys learning this sort of thing, you'll have a good time. If you're not, don't do it!

Re:After self-hosting my email for twenty-three years I have... by kagronick on Sunday September 04, 2022 @10:14PM (#32719783)
I think the problem was this guy was providing email accounts for other people. Other people who probably reused their passwords and had their email in the same DB in the apps they used. That DB was compromised. Hackers got an account to send spam. Then the domain was blacklisted.

He alluded to this in

> At some point your IP range is bound to be banned, either by one asshole IP neighbor sending spam, one of your users being pwned

Re:After self-hosting my email for twenty-three years I have... by carlesfe on Monday September 05, 2022 @04:54AM (#32721983)
I understood that sentence as something that can happen to you even if you believe it won't.

> Please believe me. My current email server IP has been managed by me and used exclusively for personal email with zero spam, zero, for the last ten years.

Re:After self-hosting my email for twenty-three years I have... by gwd on Monday September 05, 2022 @03:33AM (#32721529)
> one of your users being pwned

Wow, good catch! He really buried the lede there -- "I kept getting blacklisted because my servers kept sending out spam." I don't know what to tell you pal -- spam is a serious problem, and if your users are sending it out, you're contributing to that. If you can't keep your users in line, other providers don't have any choice but to blacklist you.

Re:After self-hosting my email for twenty-three years I have... by nibbleshifter on Monday September 05, 2022 @07:15AM (#32723046)
Except that's not what he said happened to him.

He has that there as an example of things outside your direct control that can lead to blacklisting.

Re:After self-hosting my email for twenty-three years I have... by LMYahooTFY on Monday September 05, 2022 @12:40AM (#32720592)
> The real problem is lack of incentives. The big corps do not care about e-mail. It doesn't make money and isn't easily controllable. You can't turn it into a walled garden and lock users in. So, it gets minimal attention, and only defensive measures are developed.

Is this true? Even for consumer stuff, the only Google product besides search that seems widely popular is Gmail.

For enterprise, Microsoft is pulling in billions from office products. Without digging into their statements I know from my own experience that enterprise email services constitute a core piece of this.

For consumers, switching costs are low (mostly just inconvenient to go through all of your accounts) but not relative to the drawbacks of using corporate email providers which is zero unless you value your privacy.

Re:After self-hosting my email for twenty-three years I have... by raxxorraxor on Monday September 05, 2022 @09:51AM (#32724284)
I agree, the core of the enterprise suite of Microsoft is still very much based on domain controllers and Exchange. This enables the needs for user management and access policy configuration. Where I live self-hosting with Exchange is still the usual way too. No serious company would ever use Gmail.

I wouldn't use my corporate account for private mails though. It is used to register at services and interdict the spam.

Re:After self-hosting my email for twenty-three years I have... by Barrin92 on Sunday September 04, 2022 @05:43PM (#32717972)
>the Internet was created so that there could be many independent nodes, not so that everybody has to rely on one of several blessed providers.

any community that grows large enough needs some mechanism to manage trust, this is a universal issue. The early internet was more permissive and less differentiated simply because it was smaller.

The big corps do an alright job at managing spam given the sheer size of the problem, and more importantly you don't just need to solve spam, you need to do so economically, because for your system to stay distributed the nodes need to do the job competitively.

Given that there's intrinsic benefits to managing these things at scale that's not really realistic, in large systems you're always going to have division of labor and stratification for that reason.

Re:After self-hosting my email for twenty-three years I have... by pas on Sunday September 04, 2022 @05:55PM (#32718078)
spam is solved between the big players, they already use various feedback mechanisms ... it's just not enabled for small fish.

https://en.wikipedia.org/wiki/Feedback_loop_(email)

gmail silently drops emails (while reports smtp 250 accepted) - they could just as easily report that it's blackholed. spammers already do get through their fancy AI filters.

microsoft proactively blocks half of the world, rejects the incoming mail, and sends the sysadmin on a wild goose chase to get the IP/domain/whatever allowed. then you diligently register, and wait. and then no signal from them, and the problem still persists.

so big corps, small shops, everyone and their dog flocks to the good old microsoft/google duopoly.

and at this point if someone asks what to use for email knowing ... well, it's hard to not recommend folks to just use google workspace and have some kind of backup ready for when G bans their whole account just because.

Re:After self-hosting my email for twenty-three years I have... by Godel_unicode on Monday September 05, 2022 @01:34AM (#32720913)
Use Microsoft or Google. Use your own domain. Know how to spin up the other quickly. Spend your time on things that matter.
Re:After self-hosting my email for twenty-three years I have... by hkt on Monday September 05, 2022 @04:06AM (#32721712)
I hope at some point the routine crushing of smaller providers gains the attention of the competition and markets authority, or its equivalents. It is crazy how Google and MS use their market power in this way and get away with it.
Re:After self-hosting my email for twenty-three years I have... by 29athrowaway on Sunday September 04, 2022 @10:38PM (#32719905)
Not only they don't care about other e-mail providers, they often don't care about supporting widely used e-mail clients.
Re:After self-hosting my email for twenty-three years I have... by whiplash451 on Monday September 05, 2022 @03:59AM (#32721675)
Massive corps that have a clue about how the internet works today badly care about email.

Email is your identity in many cases (for lack of a better solution).

Email -> identity -> tracking -> advertising revenue

Re:After self-hosting my email for twenty-three years I have... by Nelson69 on Monday September 05, 2022 @09:31AM (#32724098)
After 30 years, I don't think spam is going to get solved with email.
Re:After self-hosting my email for twenty-three years I have... by mleonhard on Monday September 05, 2022 @03:11PM (#32728083)
Here are some ideas that could solve spam:

1. Require the email receiver to add the sender to their contact list, then put all other emails in "Junk". The UX of this could be quite good nowadays with smartphones and QR codes. The changes to email apps are minimal:

  - 1) When a user opens a "mailto:" URL, the email program shows the normal "send email" screen with a "Just add to Contacts" button at the top.

  - 2) Email program has a function to show a QR code with the user's "mailto:" URL.
2. Add support to the SMTP protocol to let the receiving server demand that the sender make a small "postage" payment. This will require a privacy-preserving micro-transaction system. I worked on one that doesn't use crypto [0].

Here are some ideas for improving the current system, but not actually solving spam:

3. Create a protocol for automatically reporting emails marked as spam back to their admins. Servers will only accept signed emails. Then servers can do lots of things like:

  - Mark sent emails as "Receiver flagged this as spam" and inform the user.

  - When an account begins sending spam, rate limit it, disable it pending password reset, or alert the admin.
4. Add SMTP responses that mean "rejecting because your server sends spam" and "rejecting because that user sends spam". Servers can mark sent emails as "Receiver rejected this email" and inform the user.

5. Build a shared spam reporting system that accepts only signed emails and supports searching by email address. Receivers can use it to identify compromised user accounts and reject their email. Senders use it to identify receivers acting in bad faith (reporting ham as spam). A centralized version would be straightforward. A decentralized version would be a challenging project.

6. Add support to the SMTP protocol for reporting rate limits and cooling-off times to email senders. Admins of large shared email systems can feed these metrics into a monitoring system and receive alerts of problems early.

[0] https://github.com/mleonhard/hipp

EDIT: tone

Re:After self-hosting my email for twenty-three years I have... by zapataband1 on Monday September 05, 2022 @12:50AM (#32720647)
I think the real problem is how bad most e-mail interfaces are (makes you want to use it less) or that there is not a dead simple way to do it other than a provider. It should be easy enough that I can download an app on my phone and be sending messages.

I think matrix might get rid of a lot of the infrastructure barriers, but I don't have much experience with it.