News for hackers. Stuff that matters.
Re:Ptar: Replacing .tgz for petabyte-scale S3 archives by nemothekid on Tuesday July 08, 2025 @10:09PM (#44505742)
>By contrast, S3 buckets are rarely backed up (a rather short-sighted approach for mission-critical cloud data), and even one-off archives are rarely done.

This is a complete aside, but how often are people backing up data to something other than S3? What I mean is it some piece of data is on S3, do people have a contingency for "S3 failing".

S3 is so durable in my mind now that I really only imagine having an "S3 backup" if (1) I had an existing system (e.g. tapes), or (2) I need multi-cloud redundancy. Other than that, once I assume something is in S3, I confident it's safe.

Obviously this was built over years (decades?) or reliability, and if your DRP requires alternatives, you should do them, but is anyone realistically paranoid about S3?

Re:Ptar: Replacing .tgz for petabyte-scale S3 archives by kjellsbells on Tuesday July 08, 2025 @10:39PM (#44505889)
Perhaps reframe the problem not as data loss because S3's technical infrastructure failed but because of one of the many other ways that data can get zapped or that you might need it. For example:

- Employee goes rogue and nukes buckets.

- Code fault quietly deletes data, or doesnt store it like you thought.

- State entity demands access to data, and you'd rather give them a tape than your S3 keys.

I agree that with eleven-nines or whatever it is of availability, a write to S3 is not going to disappoint you, but most data losses are more about policy and personnel than infrastructure failures.

Re:Ptar: Replacing .tgz for petabyte-scale S3 archives by toomuchtodo on Tuesday July 08, 2025 @11:59PM (#44506223)
This is solved for using versioning with MFA for delete or corruption risk, S3 export if required to provide a copy. Data can also be replicated to a write only bucket in another account, with only the ability to replicate.

https://docs.aws.amazon.com/AmazonS3/latest/userguide/MultiF...

https://docs.aws.amazon.com/AmazonS3/latest/userguide/object...

Re:Ptar: Replacing .tgz for petabyte-scale S3 archives by xyzzy123 on Wednesday July 09, 2025 @12:48AM (#44506417)
Yep, for many applications, versioning is the lightweight solve.

But.. aws backup is still nice, if a bit heavy. I like common workflows to restore all stuff (ddbs, managed dbs, buckets etc) to a common point in time. Also, one of the under-appreciated causes of massive data loss is subtly incorrect lifecycle policies. Backup can save you here even when other techniques may not.

Re:Ptar: Replacing .tgz for petabyte-scale S3 archives by mrflop on Wednesday July 09, 2025 @10:43AM (#44510642)
AWS Backup can get really pricey since you pay GB-month for every single restore point. Plakar only charges once for the initial backup and then for the small deltas on whatever cheap storage you pick.

Also, AWS Backup locks your snapshots into AWS vaults, whereas Plakar lets you push and pull backups to any backend—local disk, S3, another cloud, on-prem, etc.

Re:Ptar: Replacing .tgz for petabyte-scale S3 archives by xyzzy123 on Thursday July 10, 2025 @01:29AM (#44517573)
AWS backup is a bit more nuanced than that; ideally the thing you want is N days of PITR (point in time recovery) and you want that across all your data stores (RDS dbs, buckets, dynamodb tables, etc etc), and you want to be able to restore them all to a common point in time. 7 or 30 days or PITR are common choices. It is ideal if you can perform a data restore in 1 operation since your hair may be on fire when you need to use it. In practice almost all your recovery will be from this.

The storage needed for this depends on the data change rate in your application, more or less it works like a WAL in a DB. What is annoying is that you can't really control it (for obvious reasons), and less forgivably, AWS backup is super opaque about how much is actually being used by what.

Retention of dailies / weeklies / monthlies is a different (usually compliance) concern (NOT operational, not really, if you have to restore from a monthly your business is probably already done for) and in an enterprise context you are generally prevented from using deltas for these due to enterprise policy or regulation (yeah I know it sounds crazy, reqs are getting really specific these days).

People on AWS don't generally care that they're locked in to AWS services (else.. they wouldn't be on AWS), and while cost is often a factor it is usually not the primary concern (else.. they would not be on AWS). What often IS a primary concern is knowing that their backup solution is covered under the enterprise tier AWS support they are already paying an absolute buttload for.

Also stuff like Vault lock "compliance mode" & "automated restore testing" are helpful in box-ticking scenarios.

Plakar looks awesome but I'm not sure AWS Backup customers are the right market to go for.

Re:Ptar: Replacing .tgz for petabyte-scale S3 archives by coredog64 on Tuesday July 08, 2025 @11:24PM (#44506086)
A fun one I've seen before: Your encrypted content reused a KMS key that was provisioned by a temporary CloudFormation stack and got torn down months ago.
Re:Ptar: Replacing .tgz for petabyte-scale S3 archives by foota on Wednesday July 09, 2025 @12:08AM (#44506269)
Accidental crypto shredding? Oof.
Re:Ptar: Replacing .tgz for petabyte-scale S3 archives by fpoling on Wednesday July 09, 2025 @12:52AM (#44506435)
S3 provides an object lock in compliance mode when nobody at the organization including its admins can delete objects during the specified period.
Re:Ptar: Replacing .tgz for petabyte-scale S3 archives by mrflop on Wednesday July 09, 2025 @10:54AM (#44510759)
S3 buckets can just vanish for lots of reasons. With AWS’s shared-responsibility model, you’re the one who has to back up and protect your data not AWS.
Re:Ptar: Replacing .tgz for petabyte-scale S3 archives by fpoling on Wednesday July 09, 2025 @11:38AM (#44511401)
The hold in compliance mode with AWS is accepted way to persist data that a company obliged to hold legally by US requirements.

And if your company has a sale contract with AWS the buckets cannot just vanish or AWS cannot close the account at arbitrary moment.

Re:Ptar: Replacing .tgz for petabyte-scale S3 archives by FooBarWidget on Wednesday July 09, 2025 @12:56AM (#44506453)
Or: AWS closes your account with a vague reason ("you violated our terms, we won't tell you which one") with no way to appeal.
Re:Ptar: Replacing .tgz for petabyte-scale S3 archives by joshka on Tuesday July 08, 2025 @10:17PM (#44505773)
Backups don't just protect you from durability issues. They protect you from accidental deletion, malware, and even just snapshots of what something looked at a particular time etc.

The context that this article suggests is that if your S3 bucket is your primary storage, then it's possible that you're not thinking about where the second copy of your data should belong.

Re:Ptar: Replacing .tgz for petabyte-scale S3 archives by nemothekid on Tuesday July 08, 2025 @10:26PM (#44505812)
>They protect you from accidental deletion, malware, and even just snapshots of what something looked at a particular time etc.

S3 with versioning enabled provides this. I'm not being naive when I say S3 really provides everything you might need. Its my observation over the last 13 years, dealing with tons of fires, that there has every been a situation where I couldn't retrieve something from S3.

Legally you might need an alternative. Going multi-cloud doesn't hurt - after all I do it. But practically? I don't think I would lose sleep if someone told me they only back up to S3.

Re:Ptar: Replacing .tgz for petabyte-scale S3 archives by Brian_K_White on Tuesday July 08, 2025 @11:21PM (#44506078)
And then Amazon kills your account. It doesn't matter how great their hardware and software is.
Re:Ptar: Replacing .tgz for petabyte-scale S3 archives by fpoling on Wednesday July 09, 2025 @01:11AM (#44506507)
There is a contractual obligation on Amazon side. If they kill the account in violation of the contract, the court will force them to pay heavy damages.

Now, one can argue that courts would take time and money and a company may not afford such risk even if it is theoretical. In this case if data is that important it is stupid to keep them at AWS.

But then just write the data to tapes and store in a bank cell or whatever.

Re:Ptar: Replacing .tgz for petabyte-scale S3 archives by tuckerman on Tuesday July 08, 2025 @10:39PM (#44505884)
Insider risk is a potential reason. If someone acquires root in your AWS account, having a backup might give you options to dealing with blackmail or even malicious deletion after it happens.
Re:Ptar: Replacing .tgz for petabyte-scale S3 archives by deathanatos on Tuesday July 08, 2025 @11:21PM (#44506077)
If someone acquires root in the AWS account, they likely then have access to the backups, too. Unless we're also assuming whatever is doing the backup runs in an alternate cloud and our attacker or insider somehow has access to only 1 of 2 clouds.

Possible, perhaps, but contrived.

Re:Ptar: Replacing .tgz for petabyte-scale S3 archives by coredog64 on Tuesday July 08, 2025 @11:26PM (#44506095)
There's account root and then there's org root. Accounts are security boundaries, meaning you'd want your backups to at least be in another account within the org.
Re:Ptar: Replacing .tgz for petabyte-scale S3 archives by tuckerman on Thursday July 10, 2025 @05:52PM (#44526031)
I think using a separate cloud with credentials stored in a safe (or the equivalent) isn’t that uncommon (worked places where we were nearly 100% AWS but had GCP for storing backups). You’d need to compromise/socially engineer a different set of people to get access to that.
Re:Ptar: Replacing .tgz for petabyte-scale S3 archives by icedchai on Tuesday July 08, 2025 @10:27PM (#44505817)
What if someone deletes a bucket? Then all your versioning is gone...
Re:Ptar: Replacing .tgz for petabyte-scale S3 archives by charcircuit on Tuesday July 08, 2025 @10:41PM (#44505907)
It doesn't let you.
Re:Ptar: Replacing .tgz for petabyte-scale S3 archives by icedchai on Tuesday July 08, 2025 @10:53PM (#44505957)
It can be done if you delete the versions. You’ll need to use the aws cli.
Re:Ptar: Replacing .tgz for petabyte-scale S3 archives by fpoling on Wednesday July 09, 2025 @12:57AM (#44506459)
It cannot be done if S3 objects use the object lock in compliance mode. Such objects cannot be altered in any way and the bucket cannot be deleted until the lock expires .
Re:Ptar: Replacing .tgz for petabyte-scale S3 archives by icedchai on Wednesday July 09, 2025 @04:50AM (#44507693)
Good to know! I’ve never used that feature.
Re:Ptar: Replacing .tgz for petabyte-scale S3 archives by fpoling on Wednesday July 09, 2025 @06:44AM (#44508362)
Note that with such lock mistakes can be costly. If you put into S3 several terabytes by mistake and set the compliance lock duration for 2 years, you will have to pay for that storage for 2 years.
Re:Ptar: Replacing .tgz for petabyte-scale S3 archives by icedchai on Wednesday July 09, 2025 @03:20PM (#44513842)
So not even Amazon can fix this? What if my company goes bankrupt with several TB locked up?
Re:Ptar: Replacing .tgz for petabyte-scale S3 archives by fpoling on Wednesday July 09, 2025 @04:52PM (#44514634)
If you close the account with Amazon, then yes, the data can be deleted. But typically based on contract this will require notifying Amazon and will be extremely visible and can be reverted.

If the company does not pay, then the company breaches its contract and Amazon can delete the data. But typically there would be a warning period.

Re:Ptar: Replacing .tgz for petabyte-scale S3 archives by tecleandor on Tuesday July 08, 2025 @10:14PM (#44505763)
But don't make the same mistake people make with RAID. "More durable" doesn't mean "backup".

What if somebody deletes the file? What if it got corrupted for a problem in one of your processes? What if your API key falls in the wrong hands?

Re:Ptar: Replacing .tgz for petabyte-scale S3 archives by nemothekid on Tuesday July 08, 2025 @10:24PM (#44505802)
Yes - backups also protect against someone doing a `rm -rf /*` by accident. However, I don't think I've created an S3 bucket without versioning enabled for years. If someone deletes the file, or the file gets corrupted - I just restore a pervious version.

I don't want to suggest that people should place all their eggs in one basket - it's obviously irresponsible. However, S3 (and versioning) has been the "final storage" for years now. I can only imagine a catastrophic situation like an entire s3 region blowing up. And I'm sure a disgruntled employee could do a lot of damage as well.

Re:Ptar: Replacing .tgz for petabyte-scale S3 archives by hxtk on Tuesday July 08, 2025 @10:39PM (#44505893)
I’ve worked on a project with strict legal record-keeping requirements that had a plan for the primary AWS region literally getting nuked. But that was the only contingency in our book of plans that really required the S3 backup. We generally assumed that as long as the region still existed, S3 still had everything we put in it.

Of course, since we had the backups, restoration of individual objects would’ve been possible, but we would’ve needed to do it by hand.

Re:Ptar: Replacing .tgz for petabyte-scale S3 archives by Spooky23 on Tuesday July 08, 2025 @11:30PM (#44506106)
AWS is an incredible company and S3 a best in class service. Blindly trust my business to their SLA? To every thing with write access to data? Hell, no.
Re:Ptar: Replacing .tgz for petabyte-scale S3 archives by SteveNuts on Tuesday July 08, 2025 @10:13PM (#44505759)
Yes, I am paranoid of S3. Not only could a once in a lifetime event happen, an attacker could get in and delete all my data. Data could be accidentally deleted. Corrupted data could be written...
Re:Ptar: Replacing .tgz for petabyte-scale S3 archives by nemothekid on Tuesday July 08, 2025 @10:29PM (#44505825)
>Data could be accidentally deleted. Corrupted data could be written...

You guys should really have versioning enabled. Now if someone deleted your data and all the versions, that could be possible, but that would take real effort and would like be malicious.

Re:Ptar: Replacing .tgz for petabyte-scale S3 archives by imglorp on Tuesday July 08, 2025 @10:34PM (#44505850)
Nobody mentioned the case where you get locked out of your cloud provider with no humans to speak to, or your account gets deleted by the algorithm. Both happen routinely and we only hear about it when the victim takes to the socials.
Re:Ptar: Replacing .tgz for petabyte-scale S3 archives by burnt-resistor on Tuesday July 08, 2025 @10:18PM (#44505784)
Then 3 steps.

1. Use tarsnap so there's an encryption and a management layer.

2. Use a second service so there's redundancy and no SPoF.

3. Keep cryptographic signatures (not hashes) of each backup job in something like a WORM blockchain KVS.

Re:Ptar: Replacing .tgz for petabyte-scale S3 archives by mrflop on Wednesday July 09, 2025 @10:59AM (#44510837)
That’s basically one of the reasons that led us to build Plakar.
Re:Ptar: Replacing .tgz for petabyte-scale S3 archives by zzo38computer on Tuesday July 08, 2025 @10:38PM (#44505879)
I prefer to store backups on "write once read many" media, such as DVDs. However, having multiple backups would be helpful.
Re:Ptar: Replacing .tgz for petabyte-scale S3 archives by treve on Tuesday July 08, 2025 @10:29PM (#44505827)
We can get everything back except data. It feels silly to take the risk _not_ to if you're somewhat established.
Re:Ptar: Replacing .tgz for petabyte-scale S3 archives by firesteelrain on Tuesday July 08, 2025 @10:32PM (#44505840)
My HOA uses a SmartNAS in addition to S3. And we aren’t a huge operation.
Re:Ptar: Replacing .tgz for petabyte-scale S3 archives by jamesfinlayson on Wednesday July 09, 2025 @12:41AM (#44506399)
I worked at a place that uses AWS Backup - which I assume under the hood uses S3.

The backups themselves were off-limits to regular employees though - only the team that managed AWS could edit or delete the backups.