Wait, you mean the small timers can't afford $15,000 for services like Return Path?

The entire thing is a protection racket. There are super expensive services that can get you onto those same whitelists.

I was under the impression that individual servers should be fine as long as their ip isn't on a recognized blacklist and they use free things like SPF and DMARC - is that not true?

It's not generally true unless you're a Capital-E grade Enterprise customer running a single server as a side component of a much larger Internet presence.

Finding connectivity that's not already blacklisted by the GOOG-MSFT email protection racket and will allow individual servers to send outgoing SMTP is basically impossible for sub-enterprise scale organizations. The major cloud services block outbound SMTP for non-Enterprise customers and minor cloud services will already be blacklisted.

Getting your mail delivered by GOOG-MSFT also means that you need to have matching forward and reverse DNS, and ideally you should have your own ASN that has never been associated with any addresses that have ever been accused of sending spam. This is an impossible hurdle except for large organizations.

The entire email ecosystem is stacked for force smaller players to pay a head tax either directly to GOOG-MSFT, in the form of using their mail services and paying extortionate per-user mailbox costs, or by using paid mail relay services.

So long as you do not also send spam, or send mail that someone might think is spam. I'm a mail admin and can see my users's spam reports. Lots of those messages are authenticated and there's actually not that much malicious stuff being delivered in the first place. It is just that the messages aren't wanted.

Once your messages start to get reported, the sender's reputation goes down until its spamscore qualifies for the filter.

It still possible to send mail from a small mail server, but there no grantee at all that messages will not end up in a spam folder. Most likely Gmail uses some ML based spam filter which has relatively high rate of false positives. For big senders they can do something manually, but they don't care about smaller ones. Gmail is too big and have little incentives to improve they spam filter (until users will start to leave, which will not happen anytime soon).

It's mostly true.

Yes, time for GMail et al to block them or admit it's just more anti-competitive nonsense.

If you're not whitelisted there are stricter rules and some non obvious pitfalls, but small senders with correctly configured servers does not get sent to spam in my experience.

When you pay Sendgrid or one of their competitors, you pay to not spend time dealing with this.

The overall result (in general) is to make spamming harder.

If you're not on the contact list of a personal gmail account, it's nearly certain you'll land in spam if you cold contact them. Happened to me so many times; sending a mail to the person sitting next to me who I hadn't ever contacted form my own mail server (dmarc, dkim, spf, even effing google mta-sts is configured) and I still landed in spam.

This is contrary to my experience.

The number one pitfall is that your server IPs are blacklisted (directly or by association). If you use the cheapest VPS providers they will mostly be completely blacklisted since spammers have previously used them.

We use our own email servers with our own IPs for automated emails (password resets, login tokens, welcome email, etc.) and MailChimp for newsletters. We have never had any real problem with Gmail or any other email provider.

We do see some of our clients having those problems, as far as we can tell, it is mostly caused by some sort of bad rating on IP blocks.

I mainly send from my own mail server, too. If I worry that email to a cold gmail contact will end in spam, I use a simple fallback strategy. I have gmail setup to send as an alternate email address, so it appears to come from my server's domain. The contact will certainly see gmail to gmail email, and if they reply, my email address goes into their (effective) contact list.

But is sendgrid "too big to fail"?

Too big to fail also was meant to communicate the vast amount of effort, time, and money that would be involved in rebuilding the financial infrastructure if indeed it had collapse, not to mention the immense short term pain and suffering that would occur in the interim.

A SendGrid sized void could be filled in a matter of weeks to months by teams of highly motivated engineers looking to fill a market gap. The cost of starting up there is virtually nonzero (compared to say, starting a bank), and the regulatory burden is virtually nonzero (Email regulations are arguably some of the most clear and concise).

All that to say, drawing a comparison with the "too big to fail" bailout of major US lenders is, frankly, silly.

I think you meant 'zero' not nonzero?

Yes thank you

Not even close.