I sometimes do infrastructure consulting.

One of the first questions I ask is if they have at least one fully independent, full/incremental off-site backup that can't be corrupted from the main infrastructure, and if they have ever checked if they actually work and are restorable.

I'm continuously surprised how often the answer turns out to be no after dinner digging, even in larger companies with otherwise well-run IT.

No, the automatic 7 day RDS snapshots or turning on S3 versioning is not a sufficient backup. Neither is mirroring to a S3 Glacier bucket in the same org, or rsyncing to a a backup server in the same datacenter.

Backups are annoying and unglamorous. Nobody wants to do them, or do the tedious work of validating them or setting up something like an automated restore test.

Until the day you lose your data.

> Backups are annoying and unglamorous

It gets better (as in worse) someone can easily cut down back-up expenses, and become a hero by "balancing the budget with no disruptions to operations", get a fat bonus, and then after a year or so, leave.

Their successors won't get any bonuses by increasing the budget for something that has no ROI.

And randsomware is booming!

This is a somewhat cynical take, but there are kernels of truth in there. Often, organizations not focused on quality will allow such things to happen. More than one startup has also been out out of business by not having a backup. But it’s often negligence / ignorance and not budget. When you put all the cards on the table for an organization and give them the information they need, they make much better security choices.

I think backups come up some time after MVP and launch for most organizations, this means that you don't know when it actually has to come, so someone with technical skills says hey at some point we need to put in the effort and money to get good backups and ability to restore of all this data, and someone in charge of how the business grows has to say yes we better do it.

Unless you do not open for any sort of business without having all backup system set up then you run the risk, and with thousands of companies running the risk so as to not be out-competed by faster, riskier companies someone will suffer a bad outcome of their risk-taking.

It’s negligence / ignorance that leas to stupid budget cuts.

> Their successors won't get any bonuses by increasing the budget for something that has no ROI.

Possibly, but not if the business has a culture of recognising downside risk. Single events that are not extremely unlikely and that could cripple the business are things that every company should be looking out for.

> Backups are annoying and unglamorous

Three years ago, after doing YC Startup School, I built https://www.borgbase.com to offer the simple, but secure backup service I wanted myself. Today it’s a viable business and my customers are all great and value backups as essential part of their own business. Wouldn’t want to be in any other “more glamorous” corner of the industry. Also kudos to anyone - partner or competitor - working in this “unglamorous” space. You’re all doing great work.

Do you do offsite offline backups too with verification? What if your infra gets really hacked and they wipe out all of your customer's backup data everywhere? Just because borg clients have append only modes, it doesn't stop them from deleting the raw files on your drives.

If a storage server gets p0wned, the raw data could be deleted. That's true for every cloud provider. What's important, they still can't read the backup, since it's encrypted on the client. Storage servers are also isolated from each other and in different DCs, cities and regions.

Additional offline backups aren't really feasible past a certain data volume and daily change/velocity. I'd still encourage everyone to have them for their own essential data in addition to a cloud backup. E.g. by burning it to BluRay or tape (3-2-1 rule). You can see find my own, more philosophical discussion, of the topic here: https://docs.borgbase.com/strategy/. There I also distinguish between operational backups and archives. BorgBase is focused on the former. Offline backups are more suited for the latter.

> If a storage server gets p0wned, the raw data could be deleted. That's true for every cloud provider.

It's not hard to create a cloud backup service where delete requires separate credentials which are not used in day-to-day operations (and so can be kept secure). And without these credentials a backup is kept N days and cannot be deleted or overwritten. Don't know if anyone do this, though.

You can do it with S3 policies. But that's already at the application level. The question above was about the server level below that.

How did you figure out your pricing strategy?

Prices are fairly similar in this industry. So not much to decide. Early clients did push me to add a medium plan which turned out to be popular.

Was just looking for this, will give it a go

> I'm continuously surprised how often the answer turns out to be no after dinner digging, even in larger companies with otherwise well-run IT.

I am not even remotely surprised, because most companies are not IT oriented companies, and most of those either have managers that can't be made to understand the importance of DR, or (sadly) IT people who can't.

I myself am currently in a multi-year long political battle to justify a mere $600mo to get our DR env (built with old equipment that is on the verge of irrevocable failure) moved from a closet at a branch location to an actual co-lo that won't lose power and internet 3 times a week.

At least we finally got that backup generator approved for the corporate office after the third time a goose committed suicide on our power lines. Ugh...

> No, the automatic 7 day RDS snapshots or turning on S3 versioning is not a sufficient backup. Neither is mirroring to a S3 Glacier bucket in the same org, or rsyncing to a a backup server in the same datacenter.

Why?

A few reasons come to mind.

If you lose control of an aws root account it can take weeks to get it back. That’s probably enough time for the hackers to clean out the backups.

Billing issues can lead to aws wiping out an account.

For $work the backups are in AWS but using a different payment method, account owner etc to prevent cross contamination. Honestly, they should be outside aws entirely, but separate accounts is a good start.

> If you lose control of an aws root account it can take weeks to get it back.

> Billing issues can lead to aws wiping out an account.

These are small-scale related issues though. When you're at the level where it's hard to have a reasonable backup outside of AWS, you can also resolve the root account issues fairly quickly by calling your TAM directly.

Because they share the failure domain. In all of those cases, there's a single point of failure.

This applies to average people too. I wonder who among us can say they meet your (reasonable) standard.

Like you said, backups are annoying and unglamorous. Yet, the data on my laptop is the only thing I could not replace. It's more important to me than my passport or my birth certificate. Its preservation is certainly worth a bit of thought.

> Like you said, backups are annoying and unglamorous.

It’s called having a network attached storage (NAS) device. I have a Synology NAS, which I backup to, continuously at 5 minute intervals.

Warning: Microsoft image and file backups sometimes do not work.

I recommend Acronis True Image instead, which comes with antivirus. It pretty much always works, never falter never fail. Get the version that allows you to back up to the cloud with blockchain features. You will be happy you did.

Can ransomware leak on to your always-connected NAS?

Yes, of course. This is one of the reasons why I have 3 copies of everything I use (local/computer, cloud, NAS). This is particularly why I have a cloud backup, with blockchain data authentication.

But, this is a basic overview of how to prevent NAS ransomware attacks: https://www.howtogeek.com/435452/how-to-secure-your-synology...

There is better advice elsewhere but this is a good start.

This may be a good comment to look at: https://news.ycombinator.com/item?id=25618346

This may be another good comment to take a look at: https://news.ycombinator.com/item?id=24860863

Can you elaborate some more on your cloud backup solution? More specifically, I'm curious about the privacy/security aspect. It's the main reason why I've been somewhat hesitant to adopt cloud backups for my own computers. \

I encrypt all of my information before I back it up. It is a risk that I am willing to take. Governments can always decrypt my information. Bruce Schneier, a world-renowned security expert who wrote "Click Here to Kill Everybody", states that he does not recommend storing information in foreign servers, in countries that you are not a citizen of. I suggest taking that advice.

Acronis is Swiss-based and has to comply with the GDPR, due to its direct ties with the European Union. I am a dual US|EU (Croatian) citizen, with legal rights to work/live/retire in Switzerland, so I do not have to worry about storing my information "abroad".

The EU has strict regulations, and they are only going to get stricter. There have even been talks about the EU being allowed to legally break encryption recently. If you are a third-country national (not an EU/EEA/Swiss citizen), then I do not recommend Acronis cloud backup or any other EU/EEA/Swiss service for storing your personal data. America may have its problems, and privacy may be a joke, but at least you have rights and sovereignty there.

I use rsync and hard links to create incremental backups, and Google Drive as a convenience backup. First to a collection point, then to remote locations.

It's actually managed by a side project of mine: https://github.com/nicbou/timeline

...really? What are you keeping on your laptop that is so important to you? I can't think of anything on my computer that I couldn't lose and pretty much just shrug off.

Photos, journals and other personal data.

Thank you for bringing this to light.

I have been heads down on my own side hustle as a one man show. Do you have any suggestions on how to properly approach backups for smaller groups like myself working with limited funds?

For context, some of the tech on my plate includes a few DO ubuntu droplets (hosting docker containerized services), Postgres DB (prod DO managed w/ auto-snaps, pretty much same scenario you mentioned), S3 storage for user-created assets (DO spaces w/ no backup strategy yet; no, not launched, yet), GitHub for source code, and physical MacBooks (iCloud + manual backups to a single physical external SSD).

Realistically speaking I’m bootstrapped. More importantly, I’m interested in learning the right ways in doing backups.

The horror stories I had shared with my boss are as relevant today as when they happened.

His is from the early 90s, the RAID card on the backup server was bad so it was just randomly flipping bits on the files being stored... nothing was ever tested until they needed one of those backups.

Mine was about a tape setup that followed the correct 3-2-1 setup... one of the CLI tools had an update and the agreement was waiting for user input so it would just drop and the rest of the script continued... no one had ever tested a backup so the tapes being shuffled around were empty.

A crucial point you missed:

Ransomware gangs often destroy your backup infrastructure. So it's important to create pull-only backups or backups that cannot be deleted / overwritten.

I thought this was sufficiently implied with "keep them segregated from the rest of your infrastructure," but yes, you are correct. It is important to have a set of backups that can't be destroyed by the attackers. You might get lucky that your hand-rolled solution is so hacky that a ransomware gang overlooks it. But better to not rely on that luck.

In the past, this was achieved by having a set of tapes offsite. Today, one might configure Veeam to lie when issued a delete command, and instead send the data off to an Amazon glacier instance that requires different credentials to read, write, and delete.

"Ransomware gangs often destroy your backup infrastructure. So it's important to create pull-only backups or backups that cannot be deleted / overwritten."

Every rsync.net customer has ZFS snapshots available in their account that are immutable. They are read-only.

So, even if Mallory trashes your primary site and then gains access to your rsync.net credentials, the daily/weekly/monthly snapshots cannot be destroyed.

Immutable backups are often overlooked. At borgbase.com, we call this “append-only” mode and the large majority of repositories uses it. With S3 (or similar) you would add some policies to disable deletions. So it’s usually doable, but needs to be considered when setting up the backup process.

Does the append-only mode you have in borgbase fix the issues of the append-only mode in Borg itself? https://borgbackup.readthedocs.io/en/stable/usage/notes.html... Because the way it works in Borg isn't really workable in practice, it only seems to be good to check a box.

We use the same public version of Borg, so those limitations apply. It’s still quite workable. Just prune from a different machine or once a year when the repo gets too large.

We may move to reject destructive commands from append-only repos in the future, but this will break existing workflows. Discussion on this here: https://feedback.borgbase.com/posts/28/reject-delete-and-pru...

That’s all well and good for weather data, but if anything in your backup is in any way related to user behaviors or transactions, immutability is a crime.

Immutability in this context doesn't mean "kept forever", it just means the permission to hard-delete is separate from soft-delete.

GDPR requires that good data-stewards keep backups:

the controller and the processor shall implement... the ability to restore the availability and access to personal data in a timely manner in the event of a physical or technical incident

https://gdpr-info.eu/art-32-gdpr/

For more details on GDPR and backups, see https://www.backup-systems.co.uk/blog/6-gdpr-implications-on...

It would be inappropriate, for instance, to store user data on the blockchain.

It also requires that user data can be hard-deleted on command. The ability to wipe data from backups needs to be kept close at hand.

> backups that cannot be deleted / overwritten

That gets complex if your database contains PII. If a user asks for their account to be deleted...

It's not "cannot be overwritten ever". It's more "you can overwrite it using credentials stored in a safe location and never from the production environment".

Someone will always have permissions to remove or change the entries, just not easily and not in an automated daily process way.

Would it be sufficient that the delete accounts script be managed and merged at restoration time?

The neatest way I heard is to store PII in an encrypted form and then delete encryption keys when a user requests deletion. That requires the keys to be in a different backup pipeline though.

You still have their data and anyone who bypasses the restoration script can get it so no.

This is not true. The prevailing opinion is that backups do not need to be modified and that it’s enough to delete at restore time (and to of course not process the data again).

Modern ransomware gangs focus more on data exfiltration rather than actually locking down data, and it lets them remain undetected for longer too. That said, yes, correct, having good and reliable backups is vital.

In combination with techniques to minimise easy/unrestricted exfiltration of data. Such as egress filtering, software access policies/controls, minimum privilege, network segregation/isolation, good user access and network access controls, decent patching policy, good and tested backups and incident response plan…yada yada. Make sure if they do get in, the damage is limited.

Backups help recover your data. But gangs have evolved. They also use exfiltration of your data to embarrass you into paying up, which backups do not mitigate against: the public shaming.

Why do you sweep?

https://ericalexander.org/post/devops-and-ransomware/

You know, I'm curious how large cloud providers handle this.

Obviously EC2 for AWS, but what about managed services?

A bad ransomware attack on a large cloud provider could cripple a significant portion of the internet.

VPCs are segregated.

I think we should make every sysadmin watch Mr.Robot, at the least the first season. That may drive home the point: secure backups are important. ;)

And don’t restore your cloned image using clonezilla whilst on morphine. It’s hard enough to use whilst sober.

Backups are nice. Until threat actor deletes them or encrypts them. Backups are also nice until you realize they exfiltrated data too. They are one mitigation but definitely not a sure fire insurance against a data breach and legal implications thereof.

Source: I’m a cybersecurity lawyer.

Backups should be on a different segregated network (ideally off-site) with strict access controls and encryption. The threat actor should not just be able to find them lying about (possible even directly attached) with no authentication or authorisation required to access them. If they are able to access them somehow then they shouldn’t be readable. What you describe is people with a poor backup policy, which yeah I guess describes most folk. But backups aren’t the issue here, poor backups are.

And they do not mitigate against the threat of the data being published online which seems to be the new(ish) threat these days with ransomeware.

The hardest part of getting people to do this is people dont want backups.

They want restores.