Google Cloud accidentally wiped an Australian super[annuation] (pension) fund's entire cloud deployment earlier this year. I think that if you really want durable backups, they have to be reducible to object storage and put in someone else's cloud.

From their joint statement [0]:

> when the deletion of UniSuper’s Private Cloud subscription occurred, it caused deletion across both of these geographies.

> an extensive recovery of our Private Cloud which includes hundreds of virtual machines, databases and applications.

> UniSuper had backups in place with an additional service provider. These backups have minimised data loss

Strangely enough on this last point a Google blog post [1] says:

> This incident did not impact: The customer’s data backups stored in Google Cloud Storage (GCS) in the same region.

[0] https://www.unisuper.com.au/about-us/media-centre/2024/a-joi...

[1] https://cloud.google.com/blog/products/infrastructure/detail...

they can easily go into the hundreds of TB, depending on your platform.

and i might add: i vividly remember gitlabs article how they have had automated backups and test restores for years, but when they actually needed them... it turned out some data wasn't part of it after all. just because youre testing your restore procedure doesnt mean you've actually accomplished anything.

If your restores are too time consuming to test regularly, they sure as shit aren't going to be useful in a disaster.

A newspaper that doesn't publish for a few days might recover. A bank that drops off the Swift network for days isn't a bank any more.

Banks regularly close for multiple days for bank holidays. Unscheduled downtime is a somewhat different story, though.

Luckily, the traditional SWIFT/banking infrastructure is so negligible these days, my phone can host a classic banking infrastructure for an entire small country.

A bank exists not as an isolated entitity, but as a node in a local, regional and global network of transactions.

Your phone as a “classic banking infrastructure” (nonsense phrase) can’t do credit card acquiring or realtime transactions because it’s not connected to the payment rails, transaction switches and so on (like SWIFT but all of the other less global ones run by central banks and private entities).

In developed societies, instant settlement for bank-bank transfers is the norm, and cash flow is dependent on that.

Russia and Iran pay about 2-5% for above-board (non-sanctioned) cross border transactions due to their extra costs of not being in SWIFT and USD sanctions, and between 20-50% where physical middle-people are needed to move pallets of USD.

"daily test restore is infeasible for anything but toy projects. "

It probably depends on what you call "toy" project. If you work for Google, yes I think everything is a toy project, and you're right. I only worked for ~$200M ARR/1M DAU businesses and restoring was no problem. From your point working for a FAANG business it's a toy project I can see that. But there are many more "toy projects" of this kind than FAANG companies.

"10+TB backup for example, and thats a _tiny_ backup size for a b2c product."

Sure.

I’m doing that to linux, and then the Linux box is furthermore backed up with nakivo.

Not my favorite but the price was okay and I can run the whole director on Linux, unlike all their other competitors. [veeam’s next major release 13 or 14 should do this in the next year or so too.]

While nakivo backs up s3 buckets, NFS shares, and local file servers… to your point, I don’t trust it (or any other backup software I can’t extract and unpack the resulting backup by hand) as far as I can throw it. So I rsync or mirror it to a local Linux box with aws-cli and then back THAT up.

I think you can do all this with windows stuff too but I don’t know it that well

Additionally you can take servers that are linux vps’es and do the reverse: mirror THEIR content to an s3 bucket.

You can also run minio open source/free on your fileserver and set up s3 to s3 sync. Cloudflare for example will ingest and replicate your minio server automatically and you can firewall it all off to their address ranges. It’s not free but it actually prices out favorably compared to veeam and nakivo if that’s all you need backed up.

But, backups will help if you replicated _bad data_, or more accurately _data changes_.

You can restore from backup if you accidentally ran `DELETE FROM foo;`, where replication will not help!

(Insert cryptolocker type viruses, bugs, human query mistakes, etc).

Backups are a specific point in time.

Are snapshots backups? snapshots on raid? snapshots on replicated disks?

Snapshots can be backups depending on where they are stored usually not if stored locally. For example, RDS snapshot is backup for database going down but not account being deleted or region destroyed. Generally, snapshots are way to make backups to more durable medium.

Snapshots are not backups. Snapshots on RAID are not backups. Snapshots on replicated disks are probably backups, so long as the disks being replicated to are not inside the same case/building/city/continent (pick your risk suitably) and you're not able to delete the snapshots from the machine hosting the originals.

The second SIM in my phone provides a backup for my primary service provider, so long as I keep it activated. The torch in my pocket is a backup for the lighting in my house, so long as I keep it charged. My data in tarsnap is a backup, so long as I'm able to restore it. Which means data in tarsnap isn't a complete backup on its own: unless I'm able to recover the encryption key, I don't actually have a backup.

A snapshot is a backup if a user deletes/edits their file and wants the old version. Raid is a backup if you're recovering from 1 disk failing

But yes, if the problem is simply that the main setup is down, replication will often give you a more (or even completely) up-to-date copy than a daily backup will.

backup is a replacement of specified files required by a system recovery procedure. it may be a total image, or a collection of config, and dat files, that are daily bootup settings,

> And, does restoring from backups always mean losing more _recent_ data than replication?

This depends on the archiving technology and what you're archiving.

Our file and object stores take one full backup every day. This means, we could lose up to 24 hours of data changes on these stores if something happens within these 24 hours. If this is acceptable or not depends on the RPO - the recovery point objective, or the "maximum acceptable data loss". However, especially for documents, 24 hours can be acceptable, because users and customers do tend to have files they uploaded to the system around for a few days. Especially if you have a chance to identify the lost documents.

Both on MySQL with the InnoDB driver, as well as on postgres, you can use PITR backup solutions - point in time recovery. With this, pgbackrest or e.g. xtrabackup store a full backup of the database usually once a day at our place, and then keep archiving the WAL / transaction logs of the system. And we, in turn, archive snapshots of these into the longterm archiving once a day.

If we need a restore, we'd first restore a pgbackrest or xtrabackup state from the long term archiving onto a system. And then we can use the PITR recovery mechanisms to restore at a specific point in time.

Technically, we could precisely recover down to the last transaction before the disastrous transaction to minimize data loss. In fact, I've done so one or two times after some database migrations went haywire. That involved scrolling through transaction logs with a viewer to identify when the migration tool starts running, noting down the transaction ID of the transaction tool starting it's check and then restoring to the transaction before. Very cool tbh.

This is important for an RDBMS, because the data in the relational database tends to be much more volatile than the data in a file or object store. With a filestore, users upload a file and then move it to their recycling bin or their "done" folder on the local system and can easily drag it back out tomorrow. With the database, the user spent 30 minutes to an hour writing up some text or a comment and expects it to be saved and sound once they hit "Reply". Losing this kinda data creates a lot more work & effort for our customers, because then they have to figure out what state the data is in and what to redo. This may also cause their business processes to run haywire and... it's not great.

When the hard drive failed, they restored the customer to the latest backup. Which was the tape still sitting in the tape drive in the server. It was from the first Sunday night after the system was installed years ago

Or do you mean the backup process was fine, but they restored from the wrong media, a very old tape that was about to be overwritten, instead of retrieving the one with last-week's copy?

> Regular backups seem to also only be taken once per 24 hours, though team-member-1 has not yet been able to figure out where they are stored. According to team-member-2 these don’t appear to be working, producing files only a few bytes in size.

Of course it turns out that the restore can only happen to a _new database name_ not the original, and the code had in multiple places hardcoded the assumption of what the db was called.

So restoring also involved patching the code and rolling that out; you can't "roll back" because to roll back the db the code must roll forward.

As with code deployment, it's not so scary when it's something you do so frequently that it's just a little script you run.

Yeah, I've found this to be the trick for ongoing hassle-free maintenance too. Make tearing stuff down and up frequent enough and you'll feel confident and safe when you're required to do so to recover from something.

Scariest are applications/services/servers that has been running for years but never restarted nor ever restored. Those scare me.

1) Expensive

2) Not straightforward, e.g. is there a 1:1 setup in another cloud for your system?

3) Likely to go untested and be useless when you need it most

https://rsync.net/pricing.html

That said I was once a CTO for a company with 10 photo studios and we had a large amount of new (raw, DSLR) photos per minute, so cost was an issue and also upload speed for offsite backups.

The most expensive part is going to be maintaining an up-to-date offsite data backup. Running a few VMs for a handful of hours is basically free.

But that's one helluva "just", and also means that you can't use the platform-specific features that make life easier. In practice that's probably way more expensive than spinning some testing VPSes up and down.

If Amazon decides to throw the banhammer your way, how long will it take you to retool your stack onto another cloud platform? Will your company survive if all your services are offline for a few weeks?

And if you grow beyond the startup size, can you afford being locked to proprietary technology? What are you going to do if Amazon decides to increase your prices by 100%? How are you supposed to negotiate when Amazon knows you are unable to switch to another cloud provider?

What is the most reasonable point that meets the criteria of 'as soon as possible'?

Because I imagine out of the gate doing this could be a net negative, not a net positive.

On the other hand, I'm not sufficiently well versed enough on the absolute latest devops techniques that may make this whole thing trivial, but I thought all the major cloud providers had just enough quirks in their Terraform support you can't write once standup / deploy anywhere

It's very easy to do if you don't do the absolute latest devops techniques.

Of course an accident is different than just randomly terminating service.

If you had a bunch of retailers in a shared space (like a market), and one of them was setting off fireworks, using all the power/water in the space, and scaring away customers, I'd expect them to get kicked out pretty quickly.

Now it may be that this is a false positive, I'm sure they happen, but in the case where it's a legitimate bad actor that is actively harming both the company and other customers on those servers, what's the course of action the company should take?

First they wanted us out of on-premise, and told us costs wouldn't matter.

Then they wanted us to be 'cloud agnostic', but when given deadlines changed to 'get it working in AWS ASAP, doesn't matter the tech debt'

Now they're freaking out about AWS costs, and we're back to juggling 'cloud agnostic' and 'reduce cost to serve in all clouds' priorities on top of features and maintenance, both of which are 10x slower due to tech debt and the plethora of bugs.

I really need to find a new job soon. Its insane how badly the execs and upper management are running this company. Every day is a knee jerk reaction from someone so detached from the reality of things or with so little understanding how it works, they do nothing but add process problems that barely address the issues they think they're solving.

But it ends up costing a shitton of money to switch paradigms completely, and they don't switch paradigms completely for a number of years: If you're just migrating servers to ec2/vpc, you're doing cloud wrong.

Of course, there is the idea of cloud agnostic, or even multi region, which seems a challenge for most places.

At least with terraform, it is theoretically easier to swing configurations over to a different host.

But also, the execs are the ones making the business risk decisions. Just make sure they have the correct info to make those decisions, the. Your responsibility is done.

In a world of scarcity, just keep communicating the tech debt. Maybe occasionally propose a project to address it.

My point is that even “something meaningful” comes with tech debt. It’s like that at my current place.

Too many people get “grass is greener” syndrome and think that there is some magical company somewhere which gives everyone plenty of time to refactor everything and fix all of the tech debt and execs make fantastic business risk decisions which always benefit the employee. In a world of scarcity, that practically never happens.

Just weigh your options in the market. If it’s worth staying where you are, just realize that the employee is not responsible for making business risk decisions, only responsible for sufficiently informing those who do of the facts.

Unfortunately we're at stage they will outright ignore what they're told, and then blame engineers for not being able to do what they said they couldn't do from the start. They refuse to acknowledge their impact on creating the tech debt in the first place by poor planning and wishful but impractical timelines, so proving to them we need to tackle any part of it is a struggle without letting things degrade to the point a real customer with significant money on the line is upset enough by the state of things to tackle it.

Which ultimately means we're at the horribly dysfunctional stage of management/company growth, the question is does it continue to get worse or does the CEO eventually learn and seriously look at the effectiveness of the VP levels and make changes...

There are so many providers, and therefore examples, of physical tin being accessible in under a minute with cost:hardware ratios that blow Cloud out if the sky (pun! ha!) OVH have a server for USD $95/month (with no commitments) that can be brought up and made available 120 _seconds_ that has six 3.8GHz cores, 32GB of RAM, 2x960GB NVMe SSDs, and 1Gbit/s of UNMETERED, guaranteed bandwidth... that's absolutely insane, and that's fully managed from the hardware down, so arguments like, "bUT yoU haVe to MAintAin hardWARE!" are just not true _at all_.

On premise in my opinion needs a dedicated team managing hardware and leverage solutions to provide that as VM's/Containers/etc to teams. Another team focused on OS level security and base image, then your dev teams can effectively focus on their app and leverage the automated tools provided by the hardware and OS teams.

Cloud gives you at least half of that, or all of it depending on your approach, for a cost. There are points where the cost makes sense and times when it doesn't, and typically that changes through the life of a company. Unfortunately there is a not insignificant overhead even with current tools to maintaining a truly substrate agnostic infrastructure that can be deployed on top of multiple clouds, on-premise etc... so companies are locked in even when economics change.

Exactly. At which point, you’re essentially reinventing a cloud, usually not very well. If you have access to really good people you can pull this off, and that’s why you see so many people on HN doing the “who needs cloud” flex.

But the reality is that for most companies, managing non-trivial amounts of hardware is not a core competency, and they regularly shoot themselves in the foot by trying it.

My issue is really on the other end of that scale, where getting C-suites to recognize when owning that core competency is actually beneficial to the company even if its not the focus of the company.

I grew up around companies leveraging vertical integration at the right scales to improve costs, seeing companies go the opposite direction trading all those advantages for often never-materializing benefits is... frustrating.

It seems like many people/organizations belived that they would be rid of the whole "operations problem" once they shifted all their workloads from on-prem to cloud. They believed that they paid a full team for running cables and replacing broken fans/hard drives/PSU:s, when that aspect of on-prem is a tiny (but non-zero) amount of work.

OS level security? So, "apt update && apt upgrade", then? I mean, what else are you doing, writing patches for the kernel? Checking every line of code that runs? Are you aware of how effective SELinux and systemd containers are? Just a simple firewall at the OS level? Maybe even just using Tailscale (or the open source Headscale) to introduce zero trust access capabilities.

There's a Terraform provider for Proxmox, which is an excellent hypervisor. Making a template takes less than an hour with configuration.

You do need an Ops person for sure, but an entire _team_?

Across 10k-100k+ servers, all running services and needing to orchestrate restarting across the whole fleet, while providing 0 downtime or impact to thousands of clients with terabytes of data being processed and analyzed at any given time.

Sure whats so hard about changing a tire? Well try to do it on an 18-wheeler while its driving down the highway without any impact to its speed.

> Are you aware of how effective SELinux and systemd containers are? Just a simple firewall at the OS level?

Part of a layered and in-depth system but one that introduces complexity.

>Maybe even just using Tailscale (or the open source Headscale) to introduce zero trust access capabilities.

Tailscale in an enterprise production environment? Not going to pass any sort of security audit and probably violates a number of certifications customer require at the enterprise level for network access controls, visibility and auditing.

Just managing the git/jenkins/spinnaker/terraform infrastructure in dozens of locations deploying to and maintaining tens of thousands of servers/pods requires a 24x7 team on top of the hundreds of teams and tens of thousands of devs using it.

If you're small enough that doesn't make sense, then you might be small enough one Ops person can handle the load (One is never enough if you're smart but...), but you are dealing with a very small amount of infrastructure and services at this point.

If you "need" that many servers (and aren't Google), you've built your systems massively wrong.

You're assuming that "On premise" equates to "inside our building, in racks we've installed, using power and networking we have to manage." You're correct if that's the case for your business, but my argument is based around the idea that you can use _managed_ hosting providers of physical hardware that'll be either next door to you, in the same city, or close to your users (i.e, you're a business in Germany but your customer base is in London, so you host the servers using a London based provider.)

The idea that you have to manage hardware is greatly diminished when you consider the availability of managed providers that are dirt cheap.

Those costs were moved and ended up higher than the capex costs were to begin with which everyone expected but the decision makers (they brushed it off every time they were asked in company Q&A's). Opex margins became a major issue and the company did performative layoffs and restructuring to appease the shareholders (then re-hired ~1/3 of the laid off staff within the next 8 months because they actually needed them)

The level of 'bad decision leading to bad decision' happening is somewhere between absurd and depressing at this point.

I think this all boils down to a knee-jerk reaction culture that doesn't think about the second or third degree consequences and/pr beyond the next 2-3 years.

I manually pulled a backup of everything but jeez, not good.