It gets better (as in worse) someone can easily cut down back-up expenses, and become a hero by "balancing the budget with no disruptions to operations", get a fat bonus, and then after a year or so, leave.
Their successors won't get any bonuses by increasing the budget for something that has no ROI.
And randsomware is booming!
Unless you do not open for any sort of business without having all backup system set up then you run the risk, and with thousands of companies running the risk so as to not be out-competed by faster, riskier companies someone will suffer a bad outcome of their risk-taking.
Possibly, but not if the business has a culture of recognising downside risk. Single events that are not extremely unlikely and that could cripple the business are things that every company should be looking out for.
Three years ago, after doing YC Startup School, I built https://www.borgbase.com to offer the simple, but secure backup service I wanted myself. Today it’s a viable business and my customers are all great and value backups as essential part of their own business. Wouldn’t want to be in any other “more glamorous” corner of the industry. Also kudos to anyone - partner or competitor - working in this “unglamorous” space. You’re all doing great work.
Additional offline backups aren't really feasible past a certain data volume and daily change/velocity. I'd still encourage everyone to have them for their own essential data in addition to a cloud backup. E.g. by burning it to BluRay or tape (3-2-1 rule). You can see find my own, more philosophical discussion, of the topic here: https://docs.borgbase.com/strategy/. There I also distinguish between operational backups and archives. BorgBase is focused on the former. Offline backups are more suited for the latter.
It's not hard to create a cloud backup service where delete requires separate credentials which are not used in day-to-day operations (and so can be kept secure). And without these credentials a backup is kept N days and cannot be deleted or overwritten. Don't know if anyone do this, though.
I am not even remotely surprised, because most companies are not IT oriented companies, and most of those either have managers that can't be made to understand the importance of DR, or (sadly) IT people who can't.
I myself am currently in a multi-year long political battle to justify a mere $600mo to get our DR env (built with old equipment that is on the verge of irrevocable failure) moved from a closet at a branch location to an actual co-lo that won't lose power and internet 3 times a week.
At least we finally got that backup generator approved for the corporate office after the third time a goose committed suicide on our power lines. Ugh...
Why?
If you lose control of an aws root account it can take weeks to get it back. That’s probably enough time for the hackers to clean out the backups.
Billing issues can lead to aws wiping out an account.
For $work the backups are in AWS but using a different payment method, account owner etc to prevent cross contamination. Honestly, they should be outside aws entirely, but separate accounts is a good start.
> Billing issues can lead to aws wiping out an account.
These are small-scale related issues though. When you're at the level where it's hard to have a reasonable backup outside of AWS, you can also resolve the root account issues fairly quickly by calling your TAM directly.
Like you said, backups are annoying and unglamorous. Yet, the data on my laptop is the only thing I could not replace. It's more important to me than my passport or my birth certificate. Its preservation is certainly worth a bit of thought.
It’s called having a network attached storage (NAS) device. I have a Synology NAS, which I backup to, continuously at 5 minute intervals.
Warning: Microsoft image and file backups sometimes do not work.
I recommend Acronis True Image instead, which comes with antivirus. It pretty much always works, never falter never fail. Get the version that allows you to back up to the cloud with blockchain features. You will be happy you did.
But, this is a basic overview of how to prevent NAS ransomware attacks: https://www.howtogeek.com/435452/how-to-secure-your-synology...
There is better advice elsewhere but this is a good start.
This may be a good comment to look at: https://news.ycombinator.com/item?id=25618346
This may be another good comment to take a look at: https://news.ycombinator.com/item?id=24860863
Acronis is Swiss-based and has to comply with the GDPR, due to its direct ties with the European Union. I am a dual US|EU (Croatian) citizen, with legal rights to work/live/retire in Switzerland, so I do not have to worry about storing my information "abroad".
The EU has strict regulations, and they are only going to get stricter. There have even been talks about the EU being allowed to legally break encryption recently. If you are a third-country national (not an EU/EEA/Swiss citizen), then I do not recommend Acronis cloud backup or any other EU/EEA/Swiss service for storing your personal data. America may have its problems, and privacy may be a joke, but at least you have rights and sovereignty there.
It's actually managed by a side project of mine: https://github.com/nicbou/timeline
I have been heads down on my own side hustle as a one man show. Do you have any suggestions on how to properly approach backups for smaller groups like myself working with limited funds?
For context, some of the tech on my plate includes a few DO ubuntu droplets (hosting docker containerized services), Postgres DB (prod DO managed w/ auto-snaps, pretty much same scenario you mentioned), S3 storage for user-created assets (DO spaces w/ no backup strategy yet; no, not launched, yet), GitHub for source code, and physical MacBooks (iCloud + manual backups to a single physical external SSD).
Realistically speaking I’m bootstrapped. More importantly, I’m interested in learning the right ways in doing backups.
His is from the early 90s, the RAID card on the backup server was bad so it was just randomly flipping bits on the files being stored... nothing was ever tested until they needed one of those backups.
Mine was about a tape setup that followed the correct 3-2-1 setup... one of the CLI tools had an update and the agreement was waiting for user input so it would just drop and the rest of the script continued... no one had ever tested a backup so the tapes being shuffled around were empty.
One of the first questions I ask is if they have at least one fully independent, full/incremental off-site backup that can't be corrupted from the main infrastructure, and if they have ever checked if they actually work and are restorable.
I'm continuously surprised how often the answer turns out to be no after dinner digging, even in larger companies with otherwise well-run IT.
No, the automatic 7 day RDS snapshots or turning on S3 versioning is not a sufficient backup. Neither is mirroring to a S3 Glacier bucket in the same org, or rsyncing to a a backup server in the same datacenter.
Backups are annoying and unglamorous. Nobody wants to do them, or do the tedious work of validating them or setting up something like an automated restore test.
Until the day you lose your data.