With self-hosting and SMTP, there is now a very clear TCP metadata record showing how my home IP address connected on port 25 to your home IP address. Maybe the next day you write back, and there's another nice clear metadata record showing your address connecting to me. Now any ISP between us, and anyone who can tap the ISPs' cables, knows that you and I are corresponding.
If we were both using Gmail, this metadata pattern would pretty much not appear at all: we would each make an HTTPS connection to Gmail and exchange a bunch of data, and while in principle my upload would be matched in size by your download, it would be extraordinarily noisy in many ways and hard to correlate in practice. On the other hand, Gmail would know everything about us.
Having Gmail know everything about us is clearly terrible and not a good privacy solution. However, having all of the ISPs be able to learn our detailed correspondence patterns and relationships is also clearly terrible and not a good privacy solution. So, self-hosting being a clear privacy win for messaging will require a lot of technical improvements on the metadata front. It isn't a clear win in this respect today; it most likely depends on whether you see ISPs (and people who can compromise or coerce them) or Google (and people who can compromise or coerce them) as a bigger risk overall.
Self hosting puts you in control, which means you have potentially more safety so long as you manage risk. Physical custody of your data is very valuable. The metadata thing is more of a theoretical risk.
Embrace the fact that email is not ephemeral and can end up in the wrong hands from your perspective. The other party that you're communicating with is a far more likely leak than some rogue ISP or police action.
More to the point: Google is a more central actor, who gets to see the data of far more users, than your ISP ever will.
Moreover, this moves from giving a third party your data, to giving them your metadata, and while people consistently underestimate the value of metadata, there is still an incremental improvement there.
So I ask: is there a way to provide an optional TOR endpoint? e.g. can I have an MX record with high priority that points to a .onion address, and then have a backup (with lower priority) that points to the normal global IP?
Postfix, for instance, has [transport maps](http://www.postfix.org/transport.5.html), but those let you pick a transport based on the email domain, not the MX host.
Right now, some people use [onionmx](https://github.com/ehloonion/onionmx), which maintains both a [static mapping](https://github.com/ehloonion/onionmx/blob/master/sources/map...) from domains to onion services, and a dynamic one using [SRV records](https://github.com/ehloonion/onionmx/blob/master/SRV.md).
I think the difficult part is getting outbound MTAs to actually understand that a .onion domain shouldn't be looked up over DNS.
If you're running a local tor-compatible resolver they should be!
However according to RFC 7686 https://tools.ietf.org/html/rfc7686: "Applications that do not implement the Tor protocol SHOULD generate an error upon the use of .onion and SHOULD NOT perform a DNS lookup"
Also, if ISPs are "tapped", you have a problem at the government level. IMO, privacy issues and government intrusion are not something technology can beat. Forget home servers and decentralization and usb sticks - If it's in your head, any determined government will waterboard that information out of you.
However, I tend to think of governmental privacy threats as the most important ones, and I assume that many other people in this discussion (and in the community of people who strongly advocate decentralization) share that perspective. Your prioritization is sensible if you're much more concerned with corporate monitoring (especially if you live in a jurisdiction with effective data protection legislation).
You might also want to consider people who live under extremely violent or repressive governments, whose concern with government as a threat actor may be easier to sympathize with. They're probably better off in the short term with something like Google because Google will likely resist data requests from their governments, while the ISPs (being local and sometimes state-owned) will likely not resist those requests. While they can't resist every surveillance or coercion scenario, it's easy to imagine that they can fly under the radar easily by using Gmail if it's popular where they live and they aren't doing other things online that the government finds interesting.
(This isn't purely a rhetorical question; maybe the balloon can be more tamper-evident than an undersea cable would be, or maybe the model for where link-layer or network-layer encryption can be applied could be more transparent or more controlled by a user or customer. But overall, it's tricky to explain why using RF, which there are tons of spy facilities worldwide and even in space to intercept, is safer than using an undersea cable.)
I know perfectly well, from experience, that I could vastly increase my privacy (or reduce the huge volume of information that I choose or passively accept liability for sharing, eg by having a FB account and not logging out of it every time I go to another tab etc. etc.). But how much of my time do I want to spend on opsec?
I don't want to be in a defensive posture for the rest of my life, hiding from bad actors and constantly worrying about some overlooked chink in my digital armor. Frankly, if the future is endless low-grade cyberwarfare I'd rather be on the offensive and steal other people's data for my own advantage than be endlessly upgrading my door locks.
I guess I'm just jaded because I've been through so many cycles of 'this will automate your security needs' followed by 'every system can be gamed' a couple of years later - much as Phil Zimmerman commented The natural flow of technology tends to move in the direction of making surveillance easier, and the ability of computers to track us doubles every eighteen months.
Nowadays I worry less about my privacy on the assumption that it's probably hopelessly compromised anyway, and more about being able to hold an unpopular position or opinion without official interference. For example I think it's far more important that gay people enjoy full civil rights, be able to marry etc. than they had better privacy tools that would allow them to remain in the closet indefinitely.
That's like saying that having 100% uptime and zero failures is more important than having backups. Sure the former would be ideal but in reality that answer doesn't cut it. Sure civil rights and free speech and personal autonomy are ideal, but over here in the real world there are a lot of people who still have to hide to avoid bad stuff happening to them.
I also don't think you can use 'automate' and 'security' like in the previous paragraph and expect good results.
Sadly, no, they don't. The lack of easily installed and good quality self-hosting options is probably the single biggest barrier to more people doing this. Most people don't use Google Mail because they're particularly fond of Google; they use it because it's convenient.
Running a real mail server (an SMTP host), in contrast, is certainly not something for the faint-hearted.
Running a mail store, with some sort of automatic fetching from an ISP or third party mail server and some sort of forwarding to send mail via that server, is a bit more practical for those who don't want a full-time job. Then you can provide things like IMAP or webmail access on top. Even so, it's still prohibitively difficult to actually get it set up and working reliably if you're not already a reasonably experienced system administrator, particularly if you don't have a dedicated machine to run it so you can't just install someone's turnkey package that assumes it can install or modify whatever it wants.
If you believe this is not the case, I invite you to link to some instructions that could reasonably be followed even by someone who is generally technically competent (but not an expert sysadmin) so they could host their own mail, contacts and calendar locally instead of using something like Google Mail. I suggest that no such instructions currently exist.
For contacts, calendar, just install the ownCloud or nextCloud app. On android, https://davdroid.bitfire.at/ works great.
Ever heard of mail-in-a-box[1] ?
By all means, I don't want to discourage anyone from self-hosting. The problem is: There will always be agents (private people or companies) who leak information about others (on purpose or because of a lack of concern) which allows shadow profiling of people who want to opt out. Furthermore, these agents don't necessarily grasp the dangers which the disclosure of their social network brings for them.
What we need are stronger laws as to what social networks (and other services) are allowed to store. For instance, FB shouldn't be allowed to know anything about me without me consenting to it.
I sure as hell wouldn't want that.
There are lots of hidden costs to self-hosting...
Wha...? How do you think my server receives emails while it's rebooting?
Retries on the order of a few minutes, maybe. I think I've very rarely seen maybe a handful of retries, one a day, in certain cases (thanks to mailer-daemon notifications from Gmail). I don't imagine servers keep retrying for days on end beyond that (maybe internet outage was a poor example here though). Am I wrong? How long do retries go on for? Like imagine if you were on a trip when your server crashed. Now you miss emails for two weeks and you imagine that when you reboot it after coming back the senders will keep retrying? Is that really how it works? And if it does, would you really want to have that level of availability? You never have to worry about that kind of thing with a service provider, but as soon as you maintain your own server you have to constantly monitor it and be able to fix it ASAP.