News for hackers. Stuff that matters.
Re:Announcing RDS/Aurora Snapshot Export to S3 by sandGorgon on Friday January 24, 2020 @12:35PM (#22140485)
Hmm..there's no restore? So this is like a one way export.

Also Glacier export would be nice for long term cold storage of database backups.

But a restore capability is essential.

Re:Announcing RDS/Aurora Snapshot Export to S3 by zten on Friday January 24, 2020 @12:43PM (#22140571)
The snapshots already deliver your desired functionality. This dumps it as a different format. In fact, it probably depends on restoring the snapshot to a database in order to implement this format change.

I think this is a replacement for rolling your own database exports for analytics applications with Sqoop or Spark.

Re:Announcing RDS/Aurora Snapshot Export to S3 by jungturk on Friday January 24, 2020 @03:01PM (#22141884)
Exporting to parquet has existed for a bit for AWS RDS (via AWS Data Migration Service), but this should make doing so more straightforward (since it doesn't require managing any DMS compute).

AWS DMS also supports incrementals (using the change-data-capture features in the DB).

Re:Announcing RDS/Aurora Snapshot Export to S3 by zten on Friday January 24, 2020 @03:58PM (#22142377)
Hmm, I wonder why they didn't push people towards DMS instead? This S3 export offering certainly commands a premium price for the privilege.
Re:Announcing RDS/Aurora Snapshot Export to S3 by anbotero on Friday January 24, 2020 @05:12PM (#22142951)
I just wish DMS supported real-time data synchronization for utf8mb4 on MySQL... That really destroyed my workflow with errors when I moved to utf8mb4 (requirement) and then had to stop using DMS altogether. It’s pretty much the only thing I used from that services.
Re:Announcing RDS/Aurora Snapshot Export to S3 by jungturk on Monday January 27, 2020 @10:42AM (#22160293)
There are a number of unfortunate shortcomings depending on what your DMS source/target are. Another that bit us was (the lack of) support for JSON types in MySQL sources.

Was there no option to add a transformation to your DMS routine to handle that?

Re:Announcing RDS/Aurora Snapshot Export to S3 by sandGorgon on Saturday January 25, 2020 @02:02AM (#22145159)
there is a ridiculous price difference between vanilla s3 and managed backup storage. Its close to 10x.

backup storage on postgresql is billed at $ $0.095 per GiB-month same on Aurora is billed at $0.023 per GB-month

If you include Glacier, then it is 100x. I truly believe that AWS charges these costs because they make a crazy amount of premium with that. And that is why these exports are architected to be non-restorable.

Re:Announcing RDS/Aurora Snapshot Export to S3 by nitesh_aws on Friday January 24, 2020 @01:09PM (#22140862)
Yes, that's correct, one way export (for now). Once exported you can set lifecycle policies in your S3 bucket to automatically transition to cold storage or deletion.
Re:Announcing RDS/Aurora Snapshot Export to S3 by zitterbewegung on Friday January 24, 2020 @12:39PM (#22140524)
It looks like this is more positioned to perform analysis on snapshot using other tools and not a backup solution .
Re:Announcing RDS/Aurora Snapshot Export to S3 by aquark on Friday January 24, 2020 @01:06PM (#22140825)
I’m interested in an easy backup-out-of-aws solution.

We use snapshots for regular backups, and have implemented a solutions to share and copy snapshots to a different aws account.

But given that our production database is absolutely vital to the existence of the company I’d like to have an easy way to create a secure ‘not affliated with Amazon’ copy regularly in case of the nightmare scenario where an account is compromised/terminated

Re:Announcing RDS/Aurora Snapshot Export to S3 by rsync on Friday January 24, 2020 @01:45PM (#22141201)
"I’m interested in an easy backup-out-of-aws solution."

rsync.net has 'rclone' built into the environment[1] so it is trivially easy to call rclone, over SSH:

  ssh [email protected] rclone s3:/some/bucket /rsync/account
... and perform arbitrary data movements into, out of, and between S3/glacier/nearline/gdrive resources. Your "out-of-aws solution" contains ZFS snapshots, on any schedule you like, so you can just do a "dumb" sync to rsync.net and handle the retention with the snapshot schedule.

Also, the snapshots are immutable (read-only) so your data gains some protection against ransomware and other threats like accidental deletion.

You should trust us with your backups because we've been doing this since 2001.

[1] https://rsync.net/products/rclone.html

Re:Announcing RDS/Aurora Snapshot Export to S3 by aquark on Friday January 24, 2020 @01:50PM (#22141249)
Great, thanks -- will take a look at this.

The challenge with RDS in the past has always been getting the data easily and scriptable into an S3/etc bucket to start with. This might be a good enough solution.

Restore would be a pain, but at the point its needed the pain would be acceptable.

Re:Announcing RDS/Aurora Snapshot Export to S3 by rsync on Friday January 24, 2020 @01:54PM (#22141285)
"Restore would be a pain, but at the point its needed the pain would be acceptable."

I can't speak specifically to this, but I will point out that you can just as easily run those 'rclone' commands in reverse, to move data from an rsync.net snapshot out to (insert cloud):

  ssh [email protected] rclone .zfs/snapshot/1dayago/bucket s3:/some/bucket
Re:Announcing RDS/Aurora Snapshot Export to S3 by sandGorgon on Saturday January 25, 2020 @01:59AM (#22145152)
AFAIK rsync.net doesnt work with RDS (which is a managed solution with no ssh access).
Re:Announcing RDS/Aurora Snapshot Export to S3 by rsync on Saturday January 25, 2020 @01:09PM (#22147600)
I can't speak to RDS specifically, but if you can get RDS into either S3 or Glacier, then there is an easy path to rsync.net: rclone.

The ssh access is not to Amazon, or S3 or RDS, etc. - you run the rclone command at rsync.net, over ssh:

  ssh [email protected] rclone s3:/some/bucket local/rsync.net/dir
Re:Announcing RDS/Aurora Snapshot Export to S3 by Alex3917 on Friday January 24, 2020 @01:48PM (#22141232)
> We use snapshots for regular backups, and have implemented a solutions to share and copy snapshots to a different aws account.

How do you do this if the db is encrypted? Or rather, how do you restore from the encrypted backup in the second account if the master key in the original account is somehow deleted without anyone noticing and intervening before the deletion becomes finalized?

Re:Announcing RDS/Aurora Snapshot Export to S3 by aquark on Friday January 24, 2020 @02:09PM (#22141405)
Good point ... we don't have this db encrypted as the DBMS level. Internally sensitive information is encrypted using keys not managed by KMS
Re:Announcing RDS/Aurora Snapshot Export to S3 by aeyes on Friday January 24, 2020 @12:42PM (#22140559)
The export does type conversion so this is one-way by definition :(.

https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_...

Re:Announcing RDS/Aurora Snapshot Export to S3 by joncrane on Friday January 24, 2020 @01:16PM (#22140934)
The canonical AWS answer for things that you want in Glacier that only support export to S3 is to put a lifecycle policy on the bucket to move to Glacier after 24 hours.

One day in S3 = 10 days in Glacier and since Glacier is presumably for years, the cost difference is trivial.

Re:Announcing RDS/Aurora Snapshot Export to S3 by dpedu on Friday January 24, 2020 @01:27PM (#22141039)
It is not designed to be a backup mechanism.