There is also no need to perpetuate the fear of running your own email. Decentralization of core internet services helps us all, not just the person doing it. So to anyone seriously interested and willing to learn, I highly recommend running your own email servers.
If you expect 100% delivery rate, you won't necessarily get that, no matter who is running your email infrastructure. At multiple companies, I've seen email end up in spam even though it is sent from gmail-hosted company account to gmail-hosted company account. So that's the bar you want to meet or beat and beating it is not hard.
From ~7 years experience now, I don't observe any problems with email delivery. I don't do anything special. I use postfix, it is well configured and everything works fine. I host several personal domains and a couple small business domains.
And by doing so I avoid giving google the power to cut me off from email because some arbitrary ML gone bad.
Technically maybe. But if most of the people you want to mail are on Gmail, as is the case for many, a cut off from Google is almost as bad.
I would long ago have switched over to a vendor, but I use qmail-style tagging for sub-addresses (that is, instead of user+sub@domain, I use user-sub@domain). Almost nobody supports that. Especially not GMail (where my friends could tell me that was tagged as WONTFIX).
If anybody has solved this, please do let me know (on here, via Twitter, or the email address in my bio). It's maddening.
Running the server is generally "set and forget". Every few years an issue might pop up that requires attention. These issues are generally due to a tightening of other servers' requirements rather than an actual technical issue. When such things do occur, symptoms are an occasional email rejection, and a bit of digging reveals the cause and fixing the cause returns things to normal. I've never had a wholesale rejection of mail from my server
For example, when SPF and DKIM came in, I had to add those records to my DNS. When Let's Encrypt came on-line I proactively added TLS to the server. A number of years ago lack of a Reverse DNS record got me on a blacklist for a short time. That was fixed by contacting the IP address issuer (my ISP) and getting them to add a reverse DNS record. Just make sure the reverse DNS hostname matches the hostname that your email server uses in its HELO messages. A week or so later I was automatically off the blacklist and the few rejections went away. I've never bothered with DMARC. This is the sum total of my experience with running the server.
At times there have been physical problems with the server or network outages with my ISP, but I discount these on the basis that it is my decision to run a server on a desktop PC in my house rather than in a data centre. Easily fixed if I wanted to throw money at it.
Setting everything up in the first place was a massive pain in the arse, definitely not for the feint hearted (I imagine there are tools/scripts to make it much easier nowadays), but it's been almost plain sailing since.
It's very rare that I have delivery problems, maybe once every year or two, which is roughly the same as through my O365 mailboxes!
Only issue is when there is a delivery problem, there is usually nothing you can do. Some of the block lists have a procedure for removal, many don't.
But after running my own mail server for something like 15 years, I've decided I just don't want the hassle any more (even if it isn't much work), and plan to move everything to O365.
I never get bounces from Google. And after the first day or two of removing yourself from 2 or 3 blacklists, its great.
I use mailcow FWIW
I'd encourage everyone to try this on a tiny droplet or linode with an unimportant domain (don't just migrate your entire companies' exchange into it on a Friday afternoon). It takes you an hour tops, after which you can poke around and see all the moving parts that make a good mailserver tick.
You'll also be contributing to a stronger, more resilient internet, by making it a tiny bit more decentralized.
I don't have a problem setting up a full email server, but I don't want to babysit it for just one person using it. I have too many real life obligations these days to try and fix the mess if a problem would arise.
Now try using an email from a domain name with more that 3 letters in the TLD.
Almost no one considers that a valid address.
And you'll be flagged for fraud as well.
It's very frustrating.
[0] https://www.migadu.com/en/benefits.html#anchor_catchalls
As far as I know the most important factor for deliverability is your IP address: my /24 have been clean for many years because the network operator actually respond to abuse@ reports.
If you have problems my first advice (aside from checking that SPF+DKIM+DMARC+PTR+banner+etc... are OK) would be to find a better ISP.
[1] in this block if it matters: https://rdap.arin.net/registry/ip/198.167.232.0
More, google knew this was a thing because they warned me.
(The base name also has to be different from your real email address, but that's a fairly desirable feature for disposable addresses since it doesn't reveal the real address like the '+' suffix on Gmail does.)
1. Send mail via a static IP (that I pay extra for)
2. Send all mail through a reputable 'smarthost' (mine is provided by my ISP as part of the 'business' package).
This fixes all sending mail problems. For extra fairy-dust, I added SPF records to my mail server IP.
The BIG problem is Spam. Unless you pay for an intermediary spam filtering service, you will NEVER be on top of it. After self-hosting my own mail system for decades, I am now seriously planning a migration to Office 365.
YMMV, but a combination of graylisting, RBL and "sender address valiation" tends to drop inbound spam rate to nearly zero.
My theory here is that statistically, by the time a spammer tries to send one to an active address, they're likely to have already fed things into the trainer, often multiple times.
(Disclosure: xoogler, but still happy with Gsuite...)
Out of curiosity, where does your MTA live? Do you have a VPS? VM/instance at a cloud provider? ISP with static IP? Other?
1. share everyone's mailboxes to an admin user who does the backup via IMAP.
2. create an app-password for each user which can be used to backup that user. This can be done as an admin user on your account.
Neither of them require knowing the user's password - an admin can override into each account unless it's specifically locked down to deny that. It does require a separate app password per account, we don't have a way to create a single password which can view each user's account without them explicitly sharing the folders, but I'm not sure how you reasonably do anything else without it being a backdoor behind all the privacy settings on each account.
Thanks for the tip. Maybe my Christmas present to myself will be to murder my mailserver!
Is there anything you need to enable to allow this? I just tried, and the email got bounced back.
Or are you talking about email aliases? As in, you register <user>-<tag>@<domain> first before sending emails to that address?
I chose it 20+ years ago because that's what qmail supported, back when subaddresing was a new idea. Over the ensuing years, most people ended up converging on + as the more common option, but that's only convention, not a standard. And what standard exists wasn't written until 2008: https://tools.https://tools.ietf.org/html/rfc5233ietf.org/ht...
I have never had this issue. Generally the issue is either IP reputation of your server (common with VPS providers if you get a recycled IP of a previous spammer) or your domain name.
Otherwise you are probably just unlucky enough to tickle the spam-prevention mechanisms in the almighty "algorithm" run by $BIGMAILER.
I keep one "normie" email address at a $BIGMAILER for situations like this, but at this point in my life I mostly just shrug if some big advertising/surveillance company's email system won't deliver my mail, I just won't email that person.
Be the change you want to see and all that.
> I mostly just shrug if some big advertising/surveillance company's email system won't deliver my mail, I just won't email that person.
Because that changes the tone of the opening paragraphs significantly:
> Luckily, running your own mail server is not as daunting as many would have you believe.
Sure, if you can afford to shrug off deliverability issues to major e-mail providers. Then it is, I daresay, a walk in the park!
Still, usually when sending emails to Gmail & Hotmail my emails are "lost" (they don't even appear in the spam-inbox of the receivers).
My usual workaround used to be to 1) login into my throwaway-account on their Gmail/Hotmail systems 2) send an email to my domain 3) reply to that email. After that usually my emails got accepted by those service providers, at least for a while.
I admit that nowadays I don't even do that anymore - when I see a Gmail/Hotmail recipient I just ask for another email address at a different provider or I send the email using my provider's system.
No idea if that's still the case, though.
Every time these posts about setting up a mail server reach the HN front page, there are usually complaints and they always center around third party email services. Perhaps it is the use of those services that is the problem, not the process of setting up a mail server.
Spam is probably made easier by the fact that so many people use the same third party email providers. Spammers have less servers to target. (Not the same situation if every user had their own server.) As HN commenters oft point out, email, the protocol and software, is "decentralised". But the widespread "centralised" use of the same third parties to send and deliver mail has negated the benefits of being (theoretically) decentralised.
If the FROMs top level domain is in the top 500 websites OR is a .edu NOT SPAM, ELSE, send to "proprietary spam filter". The "proprietary spam filter" is some random crap code that is unique to each email server company. You could spend years trying to figure out how to game "proprietary spam filter" - but it is a fools errand.
spam filters really are this dumb at big companies. usually the way we'd get around it is we'd call up $big_company and say, "hey, could you please remove our e-mails from your spam list? we would like to send your users a lot of mail right now." a few hours later, we'd be good to go.
this was ~2007. it may be harder to get them on the phone now, and the value of your company may need to be higher (we only had a few hundred million, which seems like chump change these days).
https://battlepenguin.com/tech/how-google-and-microsoft-made...
It's not as bad now, so I suspect someone at Google is finally listening, but I still don't trust the reliability of e-mail. If I sent someone an e-mail I haven't e-mailed before, I'll let them know on Hangouts/SMS/Facebook that I sent them an e-mail and to check their spam folder.
And if there was a secret to building a reputation you could make a lot more money as a brand consultant with it than running an email server.
I have DKIM, SPF and no relaying and the only issue in recent memory was when AT&T started bouncing me for some reason. I emailed postmaster@ saying "hey wats up," and the block was removed a day or so later.
I help with IT for a small non-profit membership org, and the problem isn't the places that throw errors when you send, but the ones that silently accept the message without complaint and route it to /dev/null. We get through fine to Google, but the other big providers are random at best about it.
"I have no trouble sending to anybody (that I've heard of from recipients" might not be as artfully written as I would like, but that's what that refers to.
Furthermore, blackholing emails like you suggest is generally rare. RFC 5321 (SMTP) states:
"[D]ropping mail without notification of the sender is permitted in practice. However, it is extremely dangerous and violates a long tradition and community expectations that mail is either delivered or returned. If silent message-dropping is misused, it could easily undermine confidence in the reliability of the Internet's mail systems. So silent dropping of messages should be considered only in those cases where there is very high confidence that the messages are seriously fraudulent or otherwise inappropriate."
The downstream MX could be configured to then send an NDR email itself but this lands you on spam lists as "backscattering" will then send spam back to whoever owns the email address the spammer is spoofing so it's generally not done.
Yes, maybe rhizome is successfully running his own personal mail server but your anecdote isn't really relevant. We can't learn from your example because the hidden rules of reputation & spam may not affect you but will affect others.
For example, a commenter (lucb1e) argued[0][1] that I was exaggerating the difficulties of reliably sending email but a year later in 2019, he confirmed the same difficulties! [2]
Do you see why we can't reliably extrapolate from personal experience?
[0] https://news.ycombinator.com/item?id=15525505
DKIM and SPF are really good signals to have. I have DKIM, SPF, DANE(nobody cares), DMARC, MTA-STS, TLS (lets encrypt) which are all potentially useful in preserving reputation and repudiating emails spoofing my domain sent from other sources.
I have never had issues getting throough to gmail. But for some mail systems, Microsoft in particular I think, they seem to penalise low volume mailers very heavily. You seem to need a certain volume for their systems to cache a reputation score and if you have a small vanity domain/mail server with very low mail volumes you may go straight to the spam folder.
Hetzner VPS, proper reverse DNS, SPF, that's all, not even DKIM. The only big ISP that doesn't like my mails is AT&T and I couldn't care less (in Europe, rarely have to deal with people @att.net).
I was running my own mail server for nearly 10 years on Hetzner. Prior to that on other hosts and in the distant past at home. Running mail servers is something I have done professionally and successfully.
The last time I moved boxes as I had many times before. I was on clean IP range but I had no IP reputation at all. In the past this wasn’t such a problem, especially with SPF, DKIM, rDNS, DMARC, server-to-server SSL etc. Around the same time I started having to deal with organisations (legal, etc for a death in the family and later rent) rather my my own circle of friends. It became extremely apparent that my mails weren’t hitting the inbox. But they were being accepted. This was extremely problematic.
I was in the group stating that running your own mail server isn’t hard. I still say that. The hard problem is convincing the big players to let your low volume domains and IPs hit the inbox. I begrudgingly gave up my MX servers last year.
Of course, there are some mails where I don't know if they landed in spam or I just didn't get a reply.
I just gave Yahoo a try, and that did go to spam, even though it was happy with both SPF and DKIM. I marked it as not spam, and then went and sent another. That went through fine.
I also tried Live. It says my SPF and DKIM are fine, but sends it to the spam box. I marked it as not spam. Nope. Next one still went to spam. Marked that as not spam and tried again. Still spam. Marked that as not spam, but didn't try sending another one yet. I'm going to wait a while, to allow for the possibility that it takes them a while to apply feedback.
I also tried comcast.net. Went through fine.
mailinabox.email is probably close to what you want, though. It's opnionated and my tip is to follow those opinions to the letter (or use another such project).
Google shows that my domain has a bad reputation, but I can't fix it because it doesn't have enough traffic. https://www.mail-tester.com gives me a 10/10 (I have DKIM, DMARC, SPF, everything), but my email still gets flagged as spam by Google.
Setting up was a bit of hassle, but since it was up, I don't do anything with it.
Occasionally I will get the odd notification that a (usually German for some reason) email address doesn't like our domain. But the major providers, never had a problem.
No idea why we haven't had a problem, I didn't know what I was doing at the time (still don't) just followed an online guide. From memory, spf, dkim which I think is all pretty basic things to setup when setting up a mail server and all guides cover it.
I've been running my own mail server without sending problems for almost a decade. Right from the beginning, I had a clue and knew that sending SMTP directly from a subscriber IP address was going to be a nonstarter. Such IP's have a negative reputation due to abuse. Instead, from the beginning, I used my ISP's (big ISP in Canada) SMTP forwarding host, configuring my SMTP server to log into that server with my credentials.
(I had a couple of hiccups along the way with the ISP's SMTP service, and they wouldn't help me unless I reproduced the issue with a supported e-mail client. Naturally, an Exim server running on Debian isn't a supported e-mail client, because it's name doesn't sound like "Microsoft Outlook" or "Mozilla Thunderbird").
Anyway, the main advantage of running your own mail server is not the ability to send SMTP directly to someone's MX server, but rather that you have your own MX domain for receiving mail, via a machine you control. Your setup isn't "lesser" in any way just because you're sending through a more reputable SMTP forwarding host.
I run a mail server for a few thousand users that has been going for ~20 years, and have regularly had to spend significant time figuring out adjustments the big mail providers make along the way. Spending a day roughly every year figuring out and implementing countermeasures isn't that bad, when you get paid for it. But I really don't have the time to do it for my personal e-mail.
Your answer reminds me of something I heard a greenskeeper say about the big garden he maintains, I think in the UK: Tourists come to me all the time and ask "How can I get my lawn to look like this?" I reply: "The secret is to roll the dew off the lawn every morning." "That's it?" "Just do that for 300 years and your lawn will look like this."
> "Just do that for 300 years and your lawn will look like this."
My point is that I've never had problems in that decade, not that it took a decade to fix. I think on two or three occasions, mail to a gmail user mysteriously bounced, but it went away on re-try.
I understood the reputation problem from the beginning instead of banging my head against it. Why I understood that is because I did some research into running mail servers before diving into it.
More to the point, I discovered this by reading Google's postmaster/bulk mail guidelines. All the other major email providers have similar guidelines, so definitely consult those as a first reference if you are having problems.
There is no mail conspiracy. As always, the problem is DNS.
What I have had an issue with is the network (the whole ####ing network, those guys are less than helpful) I’m on getting added to spamhause(I think?) and people setting up their (smaller) mail severs to blindly reject mails from networks like that. That’s really dumb and I’m glad for things like dkim because of it.
I used to run a mail server in the past, and initially had problems with reverse DNS resolution not setup correctly, it's an easy oversight depending on where you host.
https://mxtoolbox.com/problem/smtp/smtp-reverse-dns-resoluti...
Stack: OpenBSD VM (on 1984.is) running OpenSMTPd + Dovecot + a DKIM proxy + ClamAV + spamd.
The more relevant pain point is that the VM hangs every couple months and I have to hard-reboot it from the VPS web console. Also, I've been pretty lazy about staying on top of new OpenBSD releases. One of these days I'd like to go redundant (whether with both mailhosts on 1984 or putting one on a different provider), which would make it less stressful to do OS upgrades (I could keep one host running while doing an offline upgrade of the other, as opposed to hoping the online upgrade process doesn't break anything).
The only issue I have found seriously limiting my ability to use my own self hosted email server is the deliberate decision of Google, Yahoo and Microsoft to make it hard for me to do so.
Their decision is essentially un-appealable. I still get hotmail complaints about spam being sent from an IP I do not own anymore. For the life of me I could not find a way to solve this. Yes I did remove that address from their “responsible domain owner nonsense thingy” to no avail.
Often times I would send an email to a domain owned by one of the big three and that email would just disappear. No bounce back at all. The recipient would not even see it in their spam folder.
They tell you to implement various technical solution line DMARC and DKIM but in the end what really matters is the arbitrary and capricious decision of these three companies against which there is no recourse.
I agree that hosting your own email server is important but this is a battle that I gave up fighting.
Google is a different beast. Last time I remember I had to do strange things like disabling IPv6 support in my MTA (which is a good idea anyway as I'm not an expert in IPv6) in addition to the usual SPF setup etc. E-mail is delivered in general, but once in a while some letters go into spam for unknown reason. But this happens rarely, maybe a couple a times a year.
Whenever I set up a new server, I have to start from scratch though.
Right when I started with the VPS, rejections from Comcast started showing up in my logs, and I traced it back to the IP I got being in some blacklist. Requested a new IP and it has been smooth sailing ever since.
As it wasn't possible to pre-check IP addresses (it wasn't blacklisted, and in any case the email was being served from a different IP to the one that was apparently previously an issue) for any particular hosting company - and I could only afford cheap hosting - then I decided to relent and made an Outlook online account and sent the mail from that.
It seemed totally crazy to me to reject mail from a 15 year old domain that sent at most 30 emails to Outlook per month, _always_ in replay to an email; SPF/DMARC were set AFAIR and whitelisting the email address from within my own 20+yo hotmail account didn't make any difference, MS still wouldn't let _me_ receive the emails. Like if your user says the email address is fine, and sends an email to that address, .. then perhaps you should allow the user to receive that one email??
Mine goes through a DigitalOcean droplet and my mail usually gets through (based on the assumption that if I received a reply, the other person must have received it!)
By contrast a volunteer group I'm involved with was using OVH "because it's the cheapest". Suppliers and customers were routinely telling us the emails were going to spam. Moved to another provider (no idea who, the infrastructure team does that) and the complaints fell to a trickle.
Good IP block reputation (check RBLs) + valid SPF + valid DKIM is usually enough to get mail through. Except occasionally to Hotmail... but that's Hotmail...
Since mailinabox configures everything for you maybe the issue is that when you set it up yourself you don't quite get it exactly right to what gmail and microsoft demand.
I added my domain and have never had it marked as spam at any major email provider, only a few small, self-hosted servers and that was probably due to misconfiguration or high spam sensitivity. See if this might help you out.