I had some troubles with IMAP search. I set up CLucene, it was easy and enough for me (no need for Java Lucene). It just took me a long time to figure out why it wouldn't search a domain part of email addresses. It just required to set up the tokenizations in such a way to split words also on @ character, i.e. don't consider a full email address as a word. :P I also had some troubles with OpenLDAP until I finally decided to read the docs and examples there properly. Since then I have been using this setup happily and it appears I will continue to do so! I also share the LDAP with NextCloud btw.
This is a key difference. Many people who have a bad time self-hosting mail set up with a minimum-effort major provider like Hetzner or DO — you're guaranteed to have spammers as neighbours some of the time, and other networks will behave accordingly when deciding whether to accept your mail.
A real server in a proper datacentre is great — I do the same — but as a lower-cost option a virtual server with a small, clueful provider who knows their customers also works just as well.
Real hardware servers are not required to host email.
I agree, and said that in my parent comment.
A new VPS provider may work well and make you happy, but later on, many spammers may start abusing it to send spam. Then, major email providers begin to block the entire IP subnets that belong to your VPS provider and you are screwed.
Since spammers typically can't afford to buy and house servers, having your own physical or dedicated server will likely increase the likelihood of everything being fine for a long time or forever.
One example of a good signal: if you cannot sign up online, but must make personal contact with the provider (via email or other means) to set up a server. It's a minor inconvenience for you — you're only going to do it once — but spammers expect to have to do it many times and won't bother.
The point is not that much that there are such blocking lists, the point is why admins use them. Surely that's blocking a good chunk of spam, but also legit sites.
1: "As you should know now: It is not you, it is your complete provider which got [redacted] listed."
FWIW, I've been using Hetzner for the last 10 years or so (but bare metal server, not a VPS) and haven't had issues.
My recommendation is to use a provider that doesn't host spammers. Hetzner is not great at that in my experience, but sounds like they've been fine for you. Smaller providers who know their customers and are able to pay attention to how their network is being used are generally quite good at making things difficult enough for spammers that they go elsewhere.
DMARC, IP reputation and spams are not obvious.
Of course you need to learn some things, but it's not that much work. If you're the type that enjoys learning this sort of thing, you'll have a good time. If you're not, don't do it!
He alluded to this in
> At some point your IP range is bound to be banned, either by one asshole IP neighbor sending spam, one of your users being pwned
> Please believe me. My current email server IP has been managed by me and used exclusively for personal email with zero spam, zero, for the last ten years.
Wow, good catch! He really buried the lede there -- "I kept getting blacklisted because my servers kept sending out spam." I don't know what to tell you pal -- spam is a serious problem, and if your users are sending it out, you're contributing to that. If you can't keep your users in line, other providers don't have any choice but to blacklist you.
He has that there as an example of things outside your direct control that can lead to blacklisting.
Is this true? Even for consumer stuff, the only Google product besides search that seems widely popular is Gmail.
For enterprise, Microsoft is pulling in billions from office products. Without digging into their statements I know from my own experience that enterprise email services constitute a core piece of this.
For consumers, switching costs are low (mostly just inconvenient to go through all of your accounts) but not relative to the drawbacks of using corporate email providers which is zero unless you value your privacy.
I wouldn't use my corporate account for private mails though. It is used to register at services and interdict the spam.
any community that grows large enough needs some mechanism to manage trust, this is a universal issue. The early internet was more permissive and less differentiated simply because it was smaller.
The big corps do an alright job at managing spam given the sheer size of the problem, and more importantly you don't just need to solve spam, you need to do so economically, because for your system to stay distributed the nodes need to do the job competitively.
Given that there's intrinsic benefits to managing these things at scale that's not really realistic, in large systems you're always going to have division of labor and stratification for that reason.
https://en.wikipedia.org/wiki/Feedback_loop_(email)
gmail silently drops emails (while reports smtp 250 accepted) - they could just as easily report that it's blackholed. spammers already do get through their fancy AI filters.
microsoft proactively blocks half of the world, rejects the incoming mail, and sends the sysadmin on a wild goose chase to get the IP/domain/whatever allowed. then you diligently register, and wait. and then no signal from them, and the problem still persists.
so big corps, small shops, everyone and their dog flocks to the good old microsoft/google duopoly.
and at this point if someone asks what to use for email knowing ... well, it's hard to not recommend folks to just use google workspace and have some kind of backup ready for when G bans their whole account just because.
Email is your identity in many cases (for lack of a better solution).
Email -> identity -> tracking -> advertising revenue
1. Require the email receiver to add the sender to their contact list, then put all other emails in "Junk". The UX of this could be quite good nowadays with smartphones and QR codes. The changes to email apps are minimal:
- 1) When a user opens a "mailto:" URL, the email program shows the normal "send email" screen with a "Just add to Contacts" button at the top.
- 2) Email program has a function to show a QR code with the user's "mailto:" URL.
2. Add support to the SMTP protocol to let the receiving server demand that the sender make a small "postage" payment. This will require a privacy-preserving micro-transaction system. I worked on one that doesn't use crypto [0].Here are some ideas for improving the current system, but not actually solving spam:
3. Create a protocol for automatically reporting emails marked as spam back to their admins. Servers will only accept signed emails. Then servers can do lots of things like:
- Mark sent emails as "Receiver flagged this as spam" and inform the user.
- When an account begins sending spam, rate limit it, disable it pending password reset, or alert the admin.
4. Add SMTP responses that mean "rejecting because your server sends spam" and "rejecting because that user sends spam". Servers can mark sent emails as "Receiver rejected this email" and inform the user.5. Build a shared spam reporting system that accepts only signed emails and supports searching by email address. Receivers can use it to identify compromised user accounts and reject their email. Senders use it to identify receivers acting in bad faith (reporting ham as spam). A centralized version would be straightforward. A decentralized version would be a challenging project.
6. Add support to the SMTP protocol for reporting rate limits and cooling-off times to email senders. Admins of large shared email systems can feed these metrics into a monitoring system and receive alerts of problems early.
[0] https://github.com/mleonhard/hipp
EDIT: tone
I think matrix might get rid of a lot of the infrastructure barriers, but I don't have much experience with it.
Reading the comments here makes me incredibly sad. Every answer that tells me to use a provider misses the point. The Internet was created so that there could be many independent nodes, not so that everybody has to rely on one of several blessed providers. I should be able to run my own E-mail.
The real problem is lack of incentives. The big corps do not care about e-mail. It doesn't make money and isn't easily controllable. You can't turn it into a walled garden and lock users in. So, it gets minimal attention, and only defensive measures are developed.
Either we solve the spam problem, or things will get worse. The big tech corps won't solve it for us.