I was there for 10 years, and this was only a problem in the final ~2 years of my employment, which roughly lines up with your 4 years remark.
It's not about setting up perfect signed email and all the new techs. It's not about having the same clean IP for a decade. It's just the same old network effect combined with profit motive. The obvious change in the last few years (I've run a mailserver for 9) is managerial not technical.
In my humble opinion, leaving email to mega-corp alone can be a freedom risk, even if a small one.
People make a big deal of SPF/DKIM etc. I think you should, for the sake of best practice. But you can set yourself up with gsuite or Office 365 and pay no attention to these things, and you can fully expect mail to get through to everyone. You can even setup a broken SPF record that disallows office 365 email and for the most part, everyone will still get your email.
Things worked fine, but I think 2 years ago I realized that my e-mail started going to a spam folder. Fortunately adding SPF/DKIM appeared to fix it, but it feels like it is getting harder and harder now to have own mail server.
I would probably gave up, but it's infuriating to me that from courts rulings if your mail is handled by 3rd party, the 4th amendment no longer applies.
However if a domain signs up for Google Apps, magically you get whitelisted, even if you are handling your own email, funny thing that.
I guess Google's customers are just more trustworthy...
I wish. I've been having issues with being sent to spam when replying to a colleague on the same domain which is on Google. The email never left their servers, everybody was authenticated, and the emails were within an hour of each other, so it's pretty obvious that it's a reply. Still: off to spam it went.
We now have a filter rule to always consider our own domain as not spam, because it's just too unreliable otherwise.
If you get an incoming email “from” yourself and it is not marked as spam, it will be put into your sent label.
At least one person claims to have been incorrectly fired because of this. Their employer found incriminating emails in the person’s sent box, and considered it a closed case.
In that time I've had a single incident (earlier this year) where spammers got it and sent 20k spam mails in less that 24h mails. Luckily I noticed this and stopped it pretty quickly.
I had mails to some destinations bounce for around a week after that.
Beyond that incident, it's very rare to have trouble delivering mail - I think maybe once in the past 5 years there was a company who I had trouble with.
Having said all that, the spam incident this year highlighted how fragile hosting your own mail server can be if things go wrong, and I felt really bad about the spam mails sent from my server - continuing to run it almost feels like a liability, and when time permits I plan to move our mailboxes to O365
They don't refuse delivery, but that is entirely meaningless when nobody ever checks the Spam box anyways (which is also hidden by default)
Hmm, I just tried what you suggested, with 2 different accounts never sent to, and the emails got through fine.
We use SPF and DKIM, but otherwise it's a fairly standard Postfix/Dovecot install (albeit a very old one).
I have a pet theory that big email platforms care deeply about the IP "neighborhood" the sending email server lives in. So a small email server set up perfectly in Digital Ocean IP space would struggle, while an email server set up OK in (for example) CenturyLink Enterprise IP space would get through.
People here are speaking about how gmail will "effectively spambox your mails by default". That's not been my experience (from setting up multiple small customers). Anyway, at least I've never heard of gmail just eating your e-mails. They either get rejected or accepted and put somewhere (maybe the spam folder, but at least somewhere).
Office365 / hosted Exchange / Outlook Protection or whatever is it called these days... they should be routed to /dev/null by everyone.
They just won't track your reputation unless you send them more than 100 mails per day. Does this sound bad? There's more: if they don't have reputation info from you (because they refuse to track it due to the low volume), your mails will go to spam inboxes even when their filters indicate that the message is not spam.
And there's more! Don't dare to ever get a bad reputation (i.e.: a user managing to get hacked and their account used to send a couple hundred spam e-mails before triggering your countermeasures). If this happens you are 100% fucked. Now they will DROP your e-mails. Hear, hear: their servers will accept your mails and just DELETE them. No spam folder, nothing.
You will try every possible thing: set up everything for their feedback loop, sign up in their "Smart Network Data Services" to track your reputation (it will be empty except for that day)... and finally contacting them at their sender support.
Do you want to know what they will reply? That you should be patient and let your reputation build up over time. What a joke! How on earth can your reputation improve when users cannot mark your mails as "not spam" because they (outlook, not the users) are simply DELETING them without a trace?
Oh, there's a way out of all this though: obtain a "Return Path Certification" [1]. That is, pay them an absurd amount of money and your mails are guaranteed to get to the users' inbox unless you are clearly spamming (all of the above assumed you are NOT).
Up to this final point you could think they just do their best and all that I've explained is collateral damage. That last "pay and we let you off the hook" is what clearly signals to me that this is an elaborate scheme to get small players to either pay them anyway or just give up and use a big-provider service.
[1] https://sendersupport.olc.protection.outlook.com/pm/services...
It is mine. I used to have to take a proactive approach to email - send an email, and if I don't hear back in a timely manner, I check their MX records. If it's Google, I'd contact them out of band - yep... spamholed.
These days, if it's important I'll contact out of band. If not, meh... I'm not going to bother just because of Google Knows Best. I'm over pandering to Google.
For this reason one of the necessities of survival as an email sender is to classify your own outbound messages, to see if recipients might think they are spammy. If they just look spammy but you still want to send them, you need sacrificial IP addresses on separate netblocks that are dedicated to spam, so the reputation of your main IPs isn't polluted.
If there's any possibility that any of your users could get their account hijacked, or there could be malware on any device permitted to send messages, you need outbound classification.
Edit: or, alternatively, tried fighting this using anti-racketeering laws? The payment aspect sounds like a protection racket scheme to me...
I run VoIP PBXes and they occasionally need to send an email, usually for voicemail to email but occasionally for other alerts.
By default they just send straight to the internet, sending from [email protected], forward and reverse DNS mapping back to their own addresses, etc. but no other setup.
This works perfectly fine with Gmail and Gsuite accounts, but Office 365 occasionally decides it hates one of our servers. Even if the client has the sending address whitelisted, it still just gets hard blocked for no apparent reason.
Gmail, I can fire up a telnet session right now and send myself an email from my home IP address just typing raw SMTP commands in to a console. It's going to work, and unless I'm spoofing a real email address it's not even going to end up in spam.
Could you name the cost?
TLDR: Minimum tier was $400 signup + $1,375/year for up to 100,000 mails/year.
[1] https://returnpath.com/wp-content/uploads/2015/06/Return-Pat...
What was too marketing for them? A 100% opt-in list that reminded customers when a product was in stock. Fully made in-house, with captcha, privacy policies, opt-out, and with the user typing their email and name, the complete works for a one-time reminder email.
Nope far too marketing related for them.
It's harder to sign up for a Gmail account than it is to register a domain and get some hosting. And Google has their own protections against sending large amounts of mail built-in. The system is as it is now because spammers will abuse it otherwise.
The big email players are accepting mail from universities and large companies with misconfigured email servers. If you use the same rules that Google and Microsoft use to refuse emails, you will be bouncing emails from valid domains. If you don't apply them, you end up accepting spam. The big players are able to deal with the spam using specialised teams and I suspect advanced algos including machine learning.
IF the big players applied their famous rules to everybody equally, everybody, including universities, big companies etc... Would quickly configure thier mail servers properly, which WOULD reduce spam, possibly eliminate it.
Being able to keep inboxes fairly free of spam in a world full of spammers is what distinguis big email providers and enable them to sell their services.
I remember when I imagined big companies might use their considerable resources to put together something like that. Then I found out it's just a handful of random engineers cobbling together crap and hiding it behind a fancy domain name.
It is probable that Office365's entire spam filtering system is a very large and ugly spreadsheet that gets passed around between a few teams using a ticketing system.
Relevant wikipedia page: https://en.wikipedia.org/wiki/Exchange_Online_Protection
Originally, the product started out as an acquisition, and there's still some legacy code that references the old Forefront name. There's about 50 people on the team.
DKIM/DMARC/ARC is worked on by two engineers.
Right now, Google is refusing to deliver email from logwatch - giving me the "Message rejected. See https://support.google.com/mail/answer/69585 for more information." so those are going to /dev/null with no recourse to fix it because Google literally doesn't like the content of the email.
The fact that I can't hit a button and tell Google that yes indeed I'm not trying to spam myself is ridiculous.
That said, my outcoming mail traffic is almost non-existent outside of few test emails, LoL.
After setting gmail to forward everything, I noticed it was sending 33% of my legitimate email to spam.
They don’t let you disable the spam filter, but you can set up an escoteric filter that prevents it from actually putting stuff in spam.
I honestly don’t understand how people cope with gmail. Fastmail is cheap and orders of magnitude better.
I like the idea of hosting my own email but even if I get around to doing that I will still need a backup. Fastmail is probably where I will go.
In 2005 you would get 100 spams to an unfiltered, widely published address, these days it is more like 2-5.
The continued lockdowns and protocol changes of gmail and yahoo are a sign of being overstaffed, "feature" oriented and yes, the attempt to shut down competition and the free exchange of mail.
It's getting impossible to read emails, too many. If it were not for them being used to their email address and me being lazy, I would move them off to gmail.
A person from Gmail even posted on HN a while ago, stating they'll look into this and do better. That was about a year ago and the situation is exactly the same or worse.
You don't have to throw out the baby with the bathwater (:
The IMAP/POP3 server is yours, and all the email is stored on something you control.
No real issues. Sure, you make it on a spam blacklist once in a while, but at least you know when it's going to be fixed, instead of dealing with zero response (or denial) from a big hosted mail provider.
Besides, not all of "our email" comes out of our office, our CRM and ERP products send mail on our behalf, and they have problems delivering mail more than we do.
It became a self fulfilling prophecy. Our email would trigger a rule in big co's third party security app, which would then report to a centralized rule breaker clearing house automatically. Clients couldn't receive our emails, we would dive in, submit appeal to clearing house, get clear and a week later do it all again.
Everything was configured properly on our end. We passed all the online validation engines. IPs were our own personally owned block and pristine.
It became too much.
Switched to office365 and all problems magically disappeared. Sent emails to same big cos with third party email security and haven't had a single issue.