What I'd rather see is a company go into partnership with the big providers to certify individual senders at home. The idea is that you'd put up some kind of bond (a few hundred USD?) to help ensure that you won't be sending spam. In exchange for your bond, the company vouches for your IP to Google/Yahoo/Hotmail/etc and you're allowed to send your own email. After some period of use w/o any spam complaints, you get your bond money back.
I work for an ISP and run my own personal mail system. This (25/TCP outbound being blocked/restricted) is, as you mention, one of the biggest hurdles for an average person to be able to run their own mail system (especially on their home connection). I've debated offering up a cheap "outbound relay only" service for fellow geeks but I would have to (initially, at least) institute some pretty low rate restrictions in order to help prevent any spam issues. That's the biggest thing stopping me from doing it so far.
unlikely, but if you are the type to run your own mailbox, probably that matters to you
Obviously, you'd have to trust whomever you're using for relaying not to mess with your mail.
Another option could be a VPN connection to bypass ISP-level packets filters.
Been there done that. Takes a good while to build up reputation, but it's possible, even without dkim signing or anything like it. Also since it's my personal email, my policy has always been "if you can't receive it because of your mailserver, sucks to be you -- you're using that service so it's on your end."
I mentioned it's my personal email, not really meant for work (though I did use it for that also).
But technically yes, if they choose to trust Google with delivering the email and Google doesn't deliver, it's a problem on their end. Setting up your email in a way that a particular service (in this case Google) accepts it, doesn't scale.
How do you contact family and friends who aren't savvy enough to setup their own email server? What about replying to job interviews? It would really suck if you didn't get the job because the hiring manager never received your email.
aaand nearly all of them have an auto reply response with a contact email or page where you can "unblacklist" your ip easily
I've submitted this fix to RFC-Clueless several times over the past few months, but it's abandoned. Every request times out after a few weeks. Great, we're on a blacklist whose raison d'etre is "blacklist those not following the rules", and the bastards don't even follow their own declared workflow.
Most personal mail servers are not going to be sending mass emails. An algorithm that auto-spam-boxes mail from domains it's never seen before (or only receives a small trickle from) is ignorant.
An algorithm that auto-spam-boxes mail from address blocks used for consumer connections or from "new" servers is not ignorant - it is taking into account years worth of evidence that it is an effective counter-measure against spam.
I'm saying that as someone who would like to run my own home mail server, and someone who is constantly battling to make sure our (100% opt in) bulk mail servers at work don't get blocked by overzealous spam filters - I wish it wasn't this way, but it is.
I'd expect that for a spam source to be effective, it'd need to send many more messages. Some spammers may try to run many low-rate sources, but you could likely do statistical analysis on the content of the messages to differentiate the spam sources from the non-spam sources.
Further, allowing the occasional doesn't-look-like-spam-but-is-actually-spam message (if such a thing exists), isn't the end of the world. The user can mark it as spam and have the source blocked forever.
The real spam problem that needed to be addressed was mailboxes being completely flooded with unsolicited messages. So flooded that it becomes impossible to read or even find the legitimate messages.
Do you have any method by which all the different mailservers worldwide can successfully collaborate to determine precisely how many emails a particular mailserver sends per day?
You're proposing that mailservers with a low emails-per-day metric be trusted. How is that metric established? How does everyone agree on it? Do we have to implement a new protocol in addition to SMTP in order for this to be viable?
The most obvious response to this is "well each service provider can just count the stats internally and use that" but it's a non-starter. That means that Google with their, what, 1000 email servers (or more) now has to figure this out. And every other service provider too.
Finally, given all the service providers out there, and the total number of "consumer" internet connected computers, giving all the computers a free 1-10 email pass means that spam is back to the worst of 2000-2010 levels. Here's some math:
There are 115mm households in the US http://quickfacts.census.gov/qfd/states/00000.html
81% internet usage https://en.wikipedia.org/wiki/Internet_in_the_United_States#...
That's 93mm consumer connections
Let's suppose that there are 20 big players in the mail provider category, Google, Yahoo, Microsoft (hotmail, office365, etc), AOL, comcast, at&t, time warner, etc. 20 seems like a good number.
Since these systems can't be made to communicate with one another easily about the sent-count of every IP address in the world, they just do it locally. That means that the virus that infected your computer and turned it into a spam-bot gets:
10 free messages * 20 major providers * 93mm computers = 18 billion spams per day
And that's just from the computers in the US. Once you take this global, you're probably talking at least 100 major providers and 300-500 million computers. Then you're talking something like a trillion spams a day.
Seems like a reasonable idea until you look at how it would actually be implemented. Then it doesn't seem so great.
You don't need to auto-spam-box sources that are sending low volume emails to the same set of addresses, especially when the messages don't get categorized as spam by the content analysis.
You don't need the kind of global coordination you are talking about. Further, whitelists don't solve the problem. Under your scenarios, you can just as easily receive low volume spam from millions of fake accounts at the big providers. You're just relying on the provider to solve your spam problem.
Yes, most of us do need to automatically block such sources because content analysis does not do a good job, and experience demonstrates that the vast majority of such messages are still spam.
> You're just relying on the provider to solve your spam problem.
Yes, we are. And you should be extremely happy they put in the effort they do, or your e-mail would be completely useless.
Not purely, no, but content analysis is typically performed after whitelist/blacklist checks. If a host connecting to my mail servers is on a blacklist, there is no content analysis because the mail will be rejected before it gets that far.
I don't get this, is this assuming all 93 million consumer connections are infected with the same malware?
It's trivial to randomly cycle through the domains you're sending to in order to deliver at a low rate to any single domain.
> Further, allowing the occasional doesn't-look-like-spam-but-is-actually-spam message (if such a thing exists), isn't the end of the world.
It is when you allow the occasional such message from each of tens of millions of compromised machines worldwide.
Recently we had to block inbound mail from most of Ukraine because hundreds of thousands of IP addresses belonging to Ukrainian ISPs were used to spam us. I started manually blocking /24's, but couldn't keep up, so we moved to blocking /16's, and even that was going too slowly to avoid affecting our systems.
Another time recently we had to block a particularly notorious Asian ISP that basically don't care that the millions of addresses in their net bocks all gets hijacked for spamming purposes (or quite possibly they do care as a result of being paid to look the other way).
This is business as usual when you operate anything but the smallest mail servers where you might get lucky and not get detected by spammers.
We still receive spam sent from malware addressed at domains of clients we haven't hosted for half a decade, that's been picked up from peoples mail clients....
This is why a lot of people decide it's easier to just blanket block all dynamic IPs. It costs a lot of money to stay on top of spam, and a lot more if you decide to try to avoid collateral damage.
> The real spam problem that needed to be addressed was mailboxes being completely flooded with unsolicited messages. So flooded that it becomes impossible to read or even find the legitimate messages.
The practices you complain about are the reason why the spam problem is now contained sufficiently for e-mail to still be usable.
Avoiding auto-spam-boxing new domains (as opposed to IP addresses), would be helpful. Any strategy that attempts to leverage multiple machines would still need to register and maintain many domain names, making such a spam strategy costly and more technically difficult. Further, an easy check for malicious users is to examine the websites at the domains suspected of spamming. Or even just WHOIS info.
There's no way to tell whether a random host on some residential ISP or VPS provider is doing personal email or spam.
>An algorithm that auto-spam-boxes mail from domains it's never seen before (or only receives a small trickle from) is ignorant.
Since blacklisting is so prevalent, spammers who aren't idiots will be sending most of their spam from different IPs. They go out harvesting unpatched VPSes and Windows botnets. The latter is patched by residential ISPs blocking port 25, the former is still a problem.
>Maybe the big mail providers need to update their strategy regarding automatic spam-boxing?
Why? Almost all legitimate email is coming from one of the big mail providers or a big-corporate Exchange server. What reason do they have to let more spam into their users' inboxes?
In any event, I found that mainstream ISPs are willing to unblock port 25 if you tell them you are running a personal mail server and have it properly configured to not relay for unauthenticated users.
Aside from that, you'll want one static IP to make your life easy so you don't need to use dynamic DNS.
I think matters such as availability of your server and configuration are surmountable challenges.
But I strongly agree with you about security. Although I really like the idea of this device, the first thing I noticed in the product web page is the use of PHP Roundcube for webmail. I have a fear of running PHP anywhere on my network. The track record [1] is way better than, say, Wordpress, but seeing things like Exec Code, even back in 2008, gives me pause.
[1] http://www.cvedetails.com/vulnerability-list/vendor_id-8905/...
Nevertheless, in my experience, ISPs will unblock port 25 for customers on request. I've only worked with a few in my years, but I've not yet encountered any resistance.
Though it isn't always just inbound 25 traffic... often it's outbound as well. Also, the larger issue is hacked/pwned machines and bots, not just open relays. Which reminds me, I need to update Tomato on my router.
Issue with running your own mail server nowadays is spam. Gmail does a decent job filtering all the crap.
Note that the State Department's score on the FISMA report was significantly worse than that of the infamously hacked Office of Personnel Management, so maybe Ms. Clinton was on to something. https://www.whitehouse.gov/sites/default/files/omb/assets/eg...
Perhaps it was something like a PUT command to append to a user's "mail" file on the recipient computer?
This seems to suggest email was always envisioned as something to be "received" rather that voluntarily "retrieved".
In retrospect, knowing what happened to email, the later idea (e.g., djb's proposal) makes more sense.
The funny part is that even though the "receive" idea is still the core part of the email process, that is not what email users do in practice.
Users "retrieve" their mail from some third party who receives it for them.
First there was "POP" then there was "webmail".
Back in the Stone Age when email was "invented" computers were too expensive for the users to own.
Each user shared the computer with other users.
Each had an email account, usually administered by the organization that owned the expensive computer.
Today computers are inexpensive and seemingly all users own one but they still do not have control of their email accounts.
Some organization is still receiving all their mail before they do. And keeping a copy of course. :)
Then, in more recent history, the domain name "business" (a monopoly run by an organization that derives its "authority" from nowhere), fueled by the popularity of the "www", became the gatekeeper to email.
When was the last time you sent an email to user@[ipaddr]? Originally if my memory of the old RFC's is correct that was how it worked. No domain name needed.
Today, if there is no purchased, monopoly-blessed "domain name", then according to the people who control users' email there is no "valid" email address. No pay, no play. Even if port 25 is open.
Assuming a user who has paid for access to the "internet" wants control of their own email (seems reasonable), then "email" is still an addtional fee.
Domestic systems tend to be in configurations that make it hard to accept inbound TCP connections. You could serve SSL on a random port and open a port using UPNP, and it will work most of the time.
It's a difficult circle to square. The most trustworthy system is one you administer yourself and manually inspect all updates, but in practice the amount of work required makes that almost impossible. If you allow the OEM to do updates they can compromise you. If you don't do updates you end up vulnerable to exploits.
The "send a reference to the message not the message" technique was part of DJB's "internet mail 2000" proposal: https://en.wikipedia.org/wiki/Internet_Mail_2000