"If it was an ex-admin he might have deleted all the backups also."

One of the very neat things about rsync.net is that your account is on a ZFS platform and you have snapshots enabled by default and the snapshots are totally immutable.

Which means that if you back up your VPS (or your VPS company) to an rsync.net account and someone owns you and owns your rsync.net credentials, the worst they can do is delete the very latest backup ... the snapshots cannot be altered.

Just saying.

So you're telling me if I have a git server (for example) on rsync.net and a newbie developer accidentally commits sensitive credentials there's no way to remove them from the git database? That's very troubling.

I think at that point you should just rotate the credentials rather than trying to erase history.

You misunderstand on several levels here ...

First, rsync.net is "cloud storage for offsite backup" - you can't run a git server[1] (or anything else) there. It's not a VPS or a web host. It's a remote unix (ZFS) filesystem that you can access over SSH.

The other point you are missing is that the ZFS snapshots I refer to are immutable as far as you are concerned. Of course we can remove them[2], and could do so at your (vetted, verified) request. Further, we don't have unlimited disk space so the snapshots rotate out (expire) over time. Every day the 7th one is removed to make room for the new "yesterday" snapshot, and so on.

The point is, an attacker can gain full access to your backups with all of the control you have ever had over them and they can't destroy/delete the snapshots. That would have helped the victim in this story immensely.

[1] You can, however, put git repos there and interact with them, using git, over ssh.

[2] Although it requires root and is an involved, manual process - which is good.

Great explanation. Aside from using standard tools, that's the best benefit I've ever seen for your business. I rank it as best since my specialty in INFOSEC is high-strength attackers and subversion (eg malicious admins). There's rarely 3rd-party services designed to technologically counter the latter. Your solution is straight-forward, too.

One concern I have is right here:

"your (vetted, verified) request."

What's that mean specifically? There's potential for attacks there. For instance, a request from several email addresses might come from computers the admin controls. Same with some 2FA's. One would have to be careful here. I got a voice idea that just passed through my head that could leverage their smart (or dumb) phones. Also maybe dedicated tokens, apps on their phone or home computer, or something that come from your company. I'm curious what you're already doing, though.

"What's that mean specifically?"

It means that the owner and founder of rsync.net stares at your request and decides how he feels about it. Then he weighs the financial security of his family and the reputation of the business firm that has become (over these last 16 years) his life's work ... and decides if one of the engineers should call you on the phone and vet your request just a little bit more ...

Aligned interests and "skin in the game" ... those are powerful things.

The intuition of a qualified, incentivized reviewer that optionally will call a person whose voice or knowledge are known a prior. Stronger than some and weaker than others.

From my understanding of the new EU General Data Protection Rules (GDPR), it will even make rsync.net unusable for most EU companies starting 2018.

They have a no-snapshots plan, which I am using.

Or, say, one gets sued over information that you're not supposed to have and a court orders you to delete all copies. How do you comply with that?

The parent poster talked about an offline backup. That is the big idea of offline (and even better, offsite) backups, that you cannot delete or destroy them from a computer, just by destroying the physical media.

An ex admin would probably have access to the physical backups too, this could have been done before he left the company.

If he was still a full employee when he wiped the servers, then theoretically yes. It rather sounded to me that he wiped the machines after he left. And off-site backups are often handled by storage companies, making it more difficult to access them.

Isn't this the time where you put stuff on tape and store it somewhere outside of physical reach? Is tape even used anymore?

Yep, I'm not involved at all but I believe my workplace does its offsite backups on tape with a service like Iron Mountain http://www.ironmountain.com/Services/Data-Management/Tape-Va...

A patient disgruntled admin would have altered the backups to write garbage to the tapes for however long the rotation cycle is, then kicked off the production melee.