> Setting up a server in any hosting environment at this point comes with the assumption that its contents can be read at any time by the operators and whoever they let in without you ever knowing about it.

How exactly is that any different than it was 6 months ago?

The difference today is that not only should I assume that Google sysadmins could read my mail, but there's now pretty convincing arguments that Google are sending all my (as a non-US person) email to the NSA - who're storing it forever "just in case it turns out useful".

I'm somewhat less concerned about rogue individual sysadmins curiously snooping on my mail, than I am about the NSA's comprehensive perpetual archive.

What? Have you been following this topic at all?

Exactly where are these convincing arguments that google is sending all e-mail to the NSA? Links? Data? Do you have anything that supports this at all?

There is the telephone stuff that was leaked, which has been well known since 2006 [0]. But, where is this miraculous evidence that google is handing everything over to the NSA?

[0] http://yahoo.usatoday.com/news/washington/2006-05-10-nsa_x.h...

Yeah, I've been following, with the point of view of someone explicitly not protected by your 4th amendment rights. (I'm assuming, possibly incorrectly, that you're a US citizen?)

As a "non-US person", I read all the "We disclose user data to government in accordance with the law" explanations from Google/Apple/Yahoo/Facebook et al - as saying "Oh, you want our data from that guy in Australia? Sure, that's 'in accordance with the law' - here's all his email… Anything else you'd like? His social graph? His contact list/address book? His Adwords clickthroughs? His Google Analytics enabled website visits? Oh look, it seems he accidentally put his GPG private key into his GoogleDrive folder - have that too!"

Do you really find it surprising that people outside the US see Snowdon's revelations and the US government's reactions to strongly suggest this is the only sensible interpretation? _All_ the denials/re-assurances we're seeing are of the form "in accordance with the law", then going on to talk about presumed protection against _domestic_ spying, and the 4th amendment rights of US citizens. None of which is reassuring from where I sit.

Though I'd agree that non-US persons have reasons to be concerned, you moved the goal posts. It may not be particularly difficult for the NSA to get your data from Google services if they decide to target you specifically, but Google has explicitly denied broadly sending over data from some significant fraction of users, as you suggested in your previous post:

http://googleblog.blogspot.com/2013/06/what.html

Maybe - but I'm reading it with the suspicion of someone who's just having been served a wake-up call on the exact nature of a friendship, and finding it's a lot less mutually respectful than expected.

I still see that "Second, we provide user data to governments only in accordance with the law." glaring out at me from the middle of that denial.

My suspicious mind now reads phrases like " … and frequently pushes back when requests are overly broad or don’t follow the correct process" and "Press reports that suggest that Google is providing open-ended access to our users’ data are false, period", as saying "so long as 'correct process' includes a FISA rubber stamp, and that '(claims of) open-ended access to our users' can only be true if it includes all domestic US users" - then sure, we can say that while still handing over _all_ data on non-US users -it's still _technically_ true.

This is for me, kinda like when someone you thought was a close friend gets married and doesn't invite you to the wedding - nothing's actually changed, but you now view everything differently. Right now I've got that "Oh, _that's_ how it is huh? I guess that explains a few things, I wish I'd known earlier…" feeling - complete with that sense of foolishness for ever having thought things were different.

It might be paranoid - or it might just be pragmatic. Either way, things _are_ different now. I just didn't get invited to The US's wedding, while Apple, Google, Yahoo, Amazon, Skype, TeamViewer, PayPal, eBay, Digital Ocean, Linode, Rackspace, Dropbox, Evernote, Trello, Freshbooks, and a bunch of others all did. Now I have to re-evaluate all those relationships too.

How paranoid is it to wonder if I can trust 1Password any more?

The issue is not what we know, but what we don't. Keep in mind, all we have is what was leaked. We only have a tiny glimpse into the massive amounts of post-9/11 surveillance. We can ascertain that the government is doing extensive amounts of surveillance on large amounts of people, and that the NSA has either refused to tell Congress the scope of the programs or outright lied about it (take a look at this (http://www.wired.com/dangerroom/2012/06/nsa-spied/) article from a year ago , and look at Clapper's "Not Wittingly" statement).

To ask for "Links? Data? Do you have anything that supports this at all?" is asking the wrong question, since the amount that is public is incredibly small (just a tiny fraction would be my guess, but all we can do is make estimates). The question is, Is it reasonable to assume that there is more surveillance? The answer is yes, and to assume that there is detailed email surveillance, either at the ISP level, through backdoors, or through direct cooperation is not unreasonable.

"The answer is yes, and to assume that there is detailed email surveillance, either at the ISP level, through backdoors, or through direct cooperation is not unreasonable."

I'd suggest _not_ assuming that might be considered negligent.

It's not just the NSA. All big countries have intelligence services and anyone with anything to hide should assume that they will try to get their hand on as much internet traffic as they can.

So e.g. if you're a US company that has a competitor in China, and you don't want the Chinese company to read your email, it's a sensible precaution to encrypt it.

The point about trusting hosting providers is an interesting one. Indeed, when renting a Virtual Private Server from a service provider, you have no choice but to trust them to keep your data safe.

This made we wonder: would it be possible to actually secure the server in such a manner that the hosting party won't have access to your stuff without your say so ?

I think you can (sort of) do this already with having something like an encrypted virtual server running on the rented VPS. Of course, I don't think this is bullet-proof and you also do have the downside of an additional layer of overhead that comes from further virtualization of your actual server(s).

A good compromise is to just run the actual machine in your house and rent a VPS for the sole purpose of installing a VPN server to provide your machine at home with access to a static IP address that doesn't have outgoing SMTP blocked. That way all the VPS is doing is forwarding data to and from your machine at home and it doesn't have access to anything your ISP wouldn't, and TLS/PGP/etc. goes a long way to help with that as well.

That should keep anyone from reading your email (assuming you and other senders are using TLS), but the next problem is that an ISP-level observer can always tell who you're corresponding with unless you use something like mix nets or Tor. And allowing that requires some cryptographic authentication method to distinguish legitimate anonymized senders from spammers without leaking the sender's identity to an intermediary. Having the sender sign the message and then encrypt the message, the signature and anything else that identifies the sender with the recipient's public key ought to do it but I'm not sure if there is any existing software that will actually do that. It might be possible using a combination of PGP to authenticate the sender and SMTP TLS over Tor to actually transmit the message, but the missing piece is for the sender authentication to be integrated with the spam filter.

No -- unless you have exclusive physical control of the machine you'll always be vulnerable.

A malicious VPS provider has access to the machine's RAM, which means they can do almost anything. For example, they could extract your SSH private key and silently decrypt all your traffic.

Is such an attack easy? No. But I could compile sshd with debug symbols and it would be. Even without them it's still possible, just very difficult. Though nothing a skilled employee of a nation-state couldn't do.

As far as I know, SSH v2 offers perfect forward secrecy, so the attacker would have to read the session key from ram, not the private key (the private key would allow the attacker to do MITM, though).

I have a dedicated server with encrypted partitions and admin backdoors turned off at ovh. So theoretically they shouldn't be able to access the running system, and if they take it down to access the partitions directly, they're encrypted so that won't work either.

That's why they take memory snapshot first, which is trivial with VPS and then pick encryption keys from it to access encrypted volumes. This is well known method and works with pure hardware machines too with physical access. It's great question when you get to server, to shut it down or leave on. If on, it could destroy data, if turned off encryption keys are gone. I think it would require some individual case analysis before deciding which one is better approach.

Isn't the pure hardware method for memory snapshots some ridiculously complex mechanism with freezing the memory and quickly transferring it to a reading device?

If you're going to pull this, you need to know in advance that it's necessary, that's not some minor thing that everyone is going to just do automatically, it's an extra, complicated step it's easy to screw up.

That said, I acknowledge the possibility of compromise in the aforementioned scenario, but once again I don't understand people who always jump to the "I am vulnerable to this narrow threat model, ergo, I should not bother to protect myself against any threat models". Especially when the measures you can take to protect yourself most likely will address the actual threat models you are liable to encounter.

"ridiculously complex mechanism" ... Not really. Anyone with the ability to stick a USB stick in a USB port and hit a power reset button can pull off this attack:

http://www.mcgrewsecurity.com/tools/msramdmp/

Hmm, that requires USB booting though, if you disable that from the BIOS and password protect it, they can't really use this method. If they pull the battery the machine needs to be turned off and so the RAM will clear.

You don't get BIOS access to low cost dedicated servers on OVH, even though they're dedicated, you can't KVM them.

mid to high range ones you can though so that might be a workable solution there.

Also, they can still rip the ram out and nitrogen it.

Interesting stuff. TRESOR might not be overkill after all.

> Isn't the pure hardware method for memory snapshots some ridiculously complex mechanism with freezing the memory and quickly transferring it to a reading device?

With Virtual Private Servers, it's trivial to do the virtualized version of this through the hypervisor.

My original statement said "dedicated server". The response said; "virtual server attack, also works with pure hardware machines". The only version of this attack that is relevant to the original statement is one that works on a dedicated server, so re-stating that it's trivial to do this against a virtual private server isn't really adding anything to the conversation.

How do you do that with a pure hardware machine with physical access without rebooting the machine? Even if you freeze the memory with liquid nitrogen and extract the data from it, that process results in the server needing rebooted afaik. Obviously I'm assuming there's no 0-day you can just enter in via the serial console or similar.

You might chill the RAM, reboot, read out the encryption keys, mirror the disks, verify that you can decrypt it -- and then leave the server off -- mail the customer a notice saying the server rebooted due to a power glitch.

At this point you have pretty much everything on hand to help you fool the customer into believing everything is ok (eg: replace the whole physical machine with a vm...).

Come to think of it -- if you know the hardware in the machine, you probably could replace the machine with a vm anyway -- just mirror the disks and go.

The right answer. With a design that focuses on simplicity ( reducing the entry points ) you can be pretty well assured that if someone attempts to gain access to your data you will likely know.

For me that is an important factor. I rather dislike the fact that in scenarios like Gmail and other providers of their ilk -- access is provided transparently to a third party.

Running your own server can give you a reasonable expectation of privacy. Even in a hosted/colocated environment if you take the time and effort.

That said, even with a dedicated server, I still use gmail, because https://github.com/etherael/phoneme

Possibly some too-cynical questions…

Do you think ovh is any less beholden to GCHQ than Google et al are to the NSA?

Do you think your encrypted partitions and turned-off admin backdoors protect you much against people with physical access to the hardware?

> Do you think ovh is any less beholden to GCHQ than Google et al are to the NSA?

OVH maybe a little more, because at least they're not under the direct auspice of the most megalomaniacal state in the world and will act according to their economic incentives in being a reliable and trustworthy service provider, which in a purely free market would align their interests with my own. However being as they are subject to various hostile state entities that have the ability to exert deleterious effects on that self interest, to the extent that they are forced to, they will cooperate with those entities.

I take action to protect myself from those threats to the greatest extent that I am able in either scenario whilst acknowledging that every theoretical threat model can never be entirely negated, coupled with the fact that at the end of the day, bothering to break through all of my measures and taking that trouble, they would find basically nothing that they were interested in.

I despise the state, sure, I'd like to see it dissolve into something like the system outlined in the machinery of freedom, sure. However in my view, taking any violent measures to expedite this process would both make me as bad as the enemy, and compromise the goal of a better world even if it actually contributed to the downfall of the state.

> Do you think your encrypted partitions and turned-off admin backdoors protect you much against people with physical access to the hardware?

"Much" more than non encrypted partitions and not turned off admin backdoors? Absolutely. Completely and utterly impenetrable to any attack by any means at all? Not likely, I can think of one off the top of my head that would work; freeze the memory, get an image, power off the system and extract the encryption keys from the image to re-mount the partitions. That's 1) not automatic 2) not easy 3) easy to screw up 4) not undetected

Great response - thanks.

Another question - have you done the thinking or got some advice about whether marginally trusting a potentially subvert-able hosting company in a jurisdiction you don't particularly trust (US, UK, and unfortunately for me, Australia) is likely to be a better or worse bet than trying to source a server/vps in a more trusted jurisdiction? (perhaps Iceland? Or am I fooling myself assuming there's anywhere "trustworthy"?)

I used to be Australian, too.

I think of it like trusting gangs in the thieves guild, there might be some gangs that are ethically more admirable than others, but at the end of the day if it comes down to it, the administrative board of the thieves guild is able to compel any of it's member organisations to act in accordance with its edicts.

So if you're going to take any action that might cause the administrative guild of that thieves board to attack you, it should be a foregone conclusion that the trust of intermediary parties beholden to that central authority is irrelevant.

Arrange your affairs accordingly. If you must deal with gangs of thieves, deal with the better ones, and if you can avoid it, don't. And if, heaven help you, you decide to take action that will paint crosshairs on your back from the central thieves guild committee, your opsec should take into account the full powers that committee is able to bring to bear upon you.

As a former employee of OVH, I can guarantee that law enforcement compliance does happen, I don't know to what extent, but having been there for the better part of two years I have seen requests from local government be held up and data handed over to authorities. Though in those cases it was child pornography accusations.

Which could mean that the only thing an unscrupulous government official needs to do in order to access one's data is hinting at implications on child pornography.

How do you handle reboots? If all of your partitions are encrypted I would guess you use a serial console to enter in the encryption password to decrypt and mount the actual OS drive?

small boot partition (basically enough to boot, get network, start ssh), all the actual data and /tmp on encrypted stores.

But do you validate that boot partition each time you reboot the system? How do you know that your computed hashsum (or similar) is actually the true one? ...

got a hash of the boot partition, validate when it comes back up.

How do you know that you're not logging in to a vm, with all the data mirrored from your physical server?

This is reasonably clever, but how do you get a full disk mirror from a server you can't get into without powering off, if you do power off, the amount of time the system is down is a tipoff to the target as to what's going on.

Assuming however it could somehow be pulled off; I guess potential ways to mitigate would be to keep a copy of the exact hardware characteristics of the system you originally have, kernel log on bootup, precise size of disks, chipsets of all the various controllers and compare when the system reboots. It would however be possible though extremely hard to get an exact duplicate of all of these at the virtualisation layer level.

You could call the virtualisation layer in the cpu requesting access to virtualise a subsystem, clock the data bus speeds... There should be some overhead from virtualisation that wasn't there when dedicated...

It's an interesting problem. I'll think more about it.

Please note that as soon as you've typed in the key needed to unlock the disk in the vm, your data is compromised. So any verification you attempt, has to be made from the unencrypted boot image...

As for time needed, assume this: The attacker sets up a physical server that can accept the same physical disks as your server (with hotswap), with an additional disk (or ssd) for booting into a hypervisor, sets parameters for this hypervisor to closely mirror that of your server. The attackers server would typically be faster than yours -- it is/can be bought some time after your server was, so this should almost be universally possible.

The attacker boots the vm server, sets everything up; then kills power to your machine, moves the disks over. Your downtime: the time it takes to move the disks (literally).

Good luck differentiating between this and legitimate downtime.

I still think reading the encryption keys from RAM after cooling them with some co2 would be the more likely attack, but it's a fun thought experiment...

Not sure if anyone will see this, this late, but someone else recently posted this presentation on Rakasha, a malware based on top of coreboot open bios and friends -- so if you know the type of server, you might just pop up the cover and switch out the BIOS...:

https://www.youtube.com/watch?v=yRpilXPv8pU

> Setting up a server in any hosting environment at this point comes with the assumption that its contents can be read at any time by the operators and whoever they let in without you ever knowing about it.

Indeed. If you want to be secure, you need to keep your email server at home, or in some other location you trust, and make sure all communication on the net is encrypted. That way, if an adversary wants your secrets, they will have to burgle your house.

> That's still a lot better than Gmail.

True, because it takes effort for GCHQ/NSA/whoever to look at your server, which they probably won't do.

> Going beyond that to something with a whole lot more encryption and less of an ability for hosting providers to read your data would really require a product dedicated to that end: that is hard to get right

That's what I'm working on.

There is a very good chance that an email server running on a "home" IP address block of any of the major ISPs will be blacklisted.

True. If your ISP won't remove your IP from the blacklist, you could probably use a commercial service (a la Amazon's SES) for outbound SMTP while running your MX and mailstore at home. That would have somewhat different privacy tradeoffs, but it's not that different if you expect most of your email recipients to be hosted elsewhere... I may try this.

Assuming you don't have any sort of console access configured and its a physical server how could they do that without letting you know via at least a reboot?

First, they could always ask your ISP to route a copy of each smtp to their servers.

But this is a bit irrelevant until you solve the social problem: I'm going to guess that >90% of your contacts use gmail/yahoo/hotmail; so the NSA already has 90% of your mail in their databases anyways- until you convince your contacts to install their own email server.