News for hackers. Stuff that matters.
Re:Announcing RDS/Aurora Snapshot Export to S3 by whalesalad on Friday January 24, 2020 @12:43PM (#22140573)
This is a welcomed addition. Would love to see a “restore to existing DB” option. Sucks having to restore a backup to a new instance all the time.
Re:Announcing RDS/Aurora Snapshot Export to S3 by otterley on Friday January 24, 2020 @01:53PM (#22141276)
(Disclaimer: I work for AWS.)

Can you tell us more about the use case? I'm not sure I understand the need here, or how it might improve your work patterns.

Also, note that this is an export feature, not a backup/restore feature. Aurora already has native backup/restore: https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide...

Re:Announcing RDS/Aurora Snapshot Export to S3 by verelo on Friday January 24, 2020 @02:00PM (#22141332)
I've run into this need. Generally, you have a DB, you just want to bring it back to a point in time. As RDS instances can be part of a bigger setup, it's not always trivial to point your application at a different instance, so re-using an existing instance can be helpful.
Re:Announcing RDS/Aurora Snapshot Export to S3 by bklyn11201 on Friday January 24, 2020 @02:15PM (#22141473)
Re:Announcing RDS/Aurora Snapshot Export to S3 by verelo on Friday January 24, 2020 @02:17PM (#22141496)
Thats a great feature, if you're on Aurora. In my case these were MySQL RDS instances created ranging from 2012 and beyond. Migration of the app to Aurora is possible (after testing), but it would be a much larger change.
Re:Announcing RDS/Aurora Snapshot Export to S3 by otterley on Friday January 24, 2020 @02:53PM (#22141832)
We recommend you do migrate - lots of great features in Aurora while remaining 100% MySQL compatible.

(EDIT: I should have said "wire and protocol compatible" as opposed to "100%". I regret the error.)

Re:Announcing RDS/Aurora Snapshot Export to S3 by verelo on Friday January 24, 2020 @03:17PM (#22142022)
Would you sign your first born over to me on that promise? Obviously I'm joking, but i have in the past got nervous that there might be issues with this claim. I.e. previously i found that the MySQL welcome banner that Aurora gave me claimed it was a very specific version, but when i tested a bug fixed in a previous MySQL version (a weird edge cases for us around transactions) it failed the test. I'm worried that while the goal is for it to be 100% compatible, the number is closer to 99.X% and that means the migration isn't a small amount of work to verify.
Re:Announcing RDS/Aurora Snapshot Export to S3 by otterley on Friday January 24, 2020 @03:34PM (#22142174)
Definitely send us a support ticket if you find a regression in Aurora/MySQL's behavior for a specific point release.
Re:Announcing RDS/Aurora Snapshot Export to S3 by verelo on Friday January 24, 2020 @03:46PM (#22142282)
That is what I did back then, i actually got a great response and the explanation of why this issue existed was valid. Not a real concern, I just don't love the way Amazon suggests "use Aurora" all the time when the real issue from the customers perspective is "Other non-Aurora RDS offerings are lacking features, and we have to keep using them - at least for now".
Re:Announcing RDS/Aurora Snapshot Export to S3 by PeterZaitsev on Friday January 24, 2020 @03:23PM (#22142075)
Aurora compatibility with MySQL is great. But as any 100% compatibility claims this one is bullshit. You can get close but you never really quite hit 100%

https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide...

The following features are supported in MySQL 5.7.12 but are currently not supported in Aurora MySQL 5.7:

Group replication plugin

Increased page size

InnoDB buffer pool loading at startup

InnoDB full-text parser plugin

Multisource replication

Online buffer pool resizing

Password validation plugin

Query rewrite plugins

Replication filtering

The CREATE TABLESPACE SQL statement

X Protocol

Re:Announcing RDS/Aurora Snapshot Export to S3 by mritun on Friday January 24, 2020 @09:04PM (#22144156)
Those are all irrelevant for Aurora - which is a fully hosted, natively replicated database.
Re:Announcing RDS/Aurora Snapshot Export to S3 by sandGorgon on Friday January 24, 2020 @05:23PM (#22143002)
This is something we face as well on the postgresql side. We don't want to migrate.

And we want to maintain large number of multi-terabyte backups (that are RESTORABLE) in Glacier for compliance reasons.

There is orders of magnitude of cost difference between snappshots and Glacier. Also i cant maintain beyond 30 days.

Re:Announcing RDS/Aurora Snapshot Export to S3 by TheGuyWhoCodes on Friday January 24, 2020 @09:18PM (#22144204)
We have a few mysql instances where we only need them for a few days a month (maybe a week). Because a stopped instance will start automatically after 7 days we delete the instance and do a final snapshot.

When we restore we have to go through the process of configuring a new instance from that snapshot. Where you have to select the security group, availability zone, set it to not public (which btw revert to yes once you select some availability zone) etc.

This process is rather error prone and honestly annoying if you do it multiple times.

Re:Announcing RDS/Aurora Snapshot Export to S3 by whalesalad on Friday January 24, 2020 @02:08PM (#22141395)
I am imagining a scenario where the instance is shut down and restored to a point-in-time snapshot. The UI suggests this is possible based on the verbage, and this is an expectation of lots of other DB management tools.

At the moment, when you try and click 'Restore to point in time' it's just taking you through the new instance wizard with the snapshot being used as the initial dataset. Screenshot: https://i.imgur.com/vBiqgJg.png -- the 'restore to point in time' is extremely misleading.

My specific uses case would be for PostgreSQL and/or MySQL, I do not use nor have I tried Aurora.

I can't get the S3 export to work either. Screenshot: https://i.imgur.com/nabbsMr.png

I need to grab a single row out of a snapshot from our staging database. So far I have restored the DB once and our QA team was able to reproduce a bug. They began testing prior to me pulling the data out, so now I am re-cloning an instance to fetch the unmolested data from that DB.

tl;dr - allow me restore a snapshot like every other piece of software that has a snapshotting component. caveat is that this will likely need to be an offline process, but as long as that is noted and clear, that is going to help a LOT of people out.

Re:Announcing RDS/Aurora Snapshot Export to S3 by otterley on Friday January 24, 2020 @02:56PM (#22141852)
> allow me restore a snapshot like every other piece of software that has a snapshotting component. caveat is that this will likely need to be an offline process, but as long as that is noted and clear, that is going to help a LOT of people out.

We have this in Aurora already. If you migrate to Aurora (which is MySQL/Postgres compatible), you'll get this functionality out of the box.

Re:Announcing RDS/Aurora Snapshot Export to S3 by whalesalad on Friday January 24, 2020 @03:04PM (#22141913)
Is there a 'clone this database, but as an aurora database, and make it a replica that can eventually become standalone' feature?

Most of my RDS usage is for my clients, so it is not always feasable to up and forklift them to a different database platform.

I would imagine though that AWS is not going to invest a lot of time and energy into improving the developer ergonomics and UX of non-Aurora databases?

I can't think of anyone who would choose Aurora right off the bat -- becuase it's one click closer to complete vendor lock-in and away from OSS.

Re:Announcing RDS/Aurora Snapshot Export to S3 by otterley on Friday January 24, 2020 @03:11PM (#22141967)
We have migration instructions in the Aurora documentation. See, for example: https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide... (MySQL) or https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide... (Postgres).

Again, both are compatible with the open-source versions, so you can export your data if you need to via the standard tooling.

Re:Announcing RDS/Aurora Snapshot Export to S3 by nitesh_aws on Friday January 24, 2020 @03:25PM (#22142103)
Sent you an email!
Re:Announcing RDS/Aurora Snapshot Export to S3 by sandGorgon on Friday January 24, 2020 @12:35PM (#22140485)
Hmm..there's no restore? So this is like a one way export.

Also Glacier export would be nice for long term cold storage of database backups.

But a restore capability is essential.

Re:Announcing RDS/Aurora Snapshot Export to S3 by zten on Friday January 24, 2020 @12:43PM (#22140571)
The snapshots already deliver your desired functionality. This dumps it as a different format. In fact, it probably depends on restoring the snapshot to a database in order to implement this format change.

I think this is a replacement for rolling your own database exports for analytics applications with Sqoop or Spark.

Re:Announcing RDS/Aurora Snapshot Export to S3 by jungturk on Friday January 24, 2020 @03:01PM (#22141884)
Exporting to parquet has existed for a bit for AWS RDS (via AWS Data Migration Service), but this should make doing so more straightforward (since it doesn't require managing any DMS compute).

AWS DMS also supports incrementals (using the change-data-capture features in the DB).

Re:Announcing RDS/Aurora Snapshot Export to S3 by zten on Friday January 24, 2020 @03:58PM (#22142377)
Hmm, I wonder why they didn't push people towards DMS instead? This S3 export offering certainly commands a premium price for the privilege.
Re:Announcing RDS/Aurora Snapshot Export to S3 by anbotero on Friday January 24, 2020 @05:12PM (#22142951)
I just wish DMS supported real-time data synchronization for utf8mb4 on MySQL... That really destroyed my workflow with errors when I moved to utf8mb4 (requirement) and then had to stop using DMS altogether. It’s pretty much the only thing I used from that services.
Re:Announcing RDS/Aurora Snapshot Export to S3 by jungturk on Monday January 27, 2020 @10:42AM (#22160293)
There are a number of unfortunate shortcomings depending on what your DMS source/target are. Another that bit us was (the lack of) support for JSON types in MySQL sources.

Was there no option to add a transformation to your DMS routine to handle that?

Re:Announcing RDS/Aurora Snapshot Export to S3 by sandGorgon on Saturday January 25, 2020 @02:02AM (#22145159)
there is a ridiculous price difference between vanilla s3 and managed backup storage. Its close to 10x.

backup storage on postgresql is billed at $ $0.095 per GiB-month same on Aurora is billed at $0.023 per GB-month

If you include Glacier, then it is 100x. I truly believe that AWS charges these costs because they make a crazy amount of premium with that. And that is why these exports are architected to be non-restorable.

Re:Announcing RDS/Aurora Snapshot Export to S3 by nitesh_aws on Friday January 24, 2020 @01:09PM (#22140862)
Yes, that's correct, one way export (for now). Once exported you can set lifecycle policies in your S3 bucket to automatically transition to cold storage or deletion.
Re:Announcing RDS/Aurora Snapshot Export to S3 by zitterbewegung on Friday January 24, 2020 @12:39PM (#22140524)
It looks like this is more positioned to perform analysis on snapshot using other tools and not a backup solution .
Re:Announcing RDS/Aurora Snapshot Export to S3 by aquark on Friday January 24, 2020 @01:06PM (#22140825)
I’m interested in an easy backup-out-of-aws solution.

We use snapshots for regular backups, and have implemented a solutions to share and copy snapshots to a different aws account.

But given that our production database is absolutely vital to the existence of the company I’d like to have an easy way to create a secure ‘not affliated with Amazon’ copy regularly in case of the nightmare scenario where an account is compromised/terminated

Re:Announcing RDS/Aurora Snapshot Export to S3 by rsync on Friday January 24, 2020 @01:45PM (#22141201)
"I’m interested in an easy backup-out-of-aws solution."

rsync.net has 'rclone' built into the environment[1] so it is trivially easy to call rclone, over SSH:

  ssh [email protected] rclone s3:/some/bucket /rsync/account
... and perform arbitrary data movements into, out of, and between S3/glacier/nearline/gdrive resources. Your "out-of-aws solution" contains ZFS snapshots, on any schedule you like, so you can just do a "dumb" sync to rsync.net and handle the retention with the snapshot schedule.

Also, the snapshots are immutable (read-only) so your data gains some protection against ransomware and other threats like accidental deletion.

You should trust us with your backups because we've been doing this since 2001.

[1] https://rsync.net/products/rclone.html

Re:Announcing RDS/Aurora Snapshot Export to S3 by aquark on Friday January 24, 2020 @01:50PM (#22141249)
Great, thanks -- will take a look at this.

The challenge with RDS in the past has always been getting the data easily and scriptable into an S3/etc bucket to start with. This might be a good enough solution.

Restore would be a pain, but at the point its needed the pain would be acceptable.

Re:Announcing RDS/Aurora Snapshot Export to S3 by rsync on Friday January 24, 2020 @01:54PM (#22141285)
"Restore would be a pain, but at the point its needed the pain would be acceptable."

I can't speak specifically to this, but I will point out that you can just as easily run those 'rclone' commands in reverse, to move data from an rsync.net snapshot out to (insert cloud):

  ssh [email protected] rclone .zfs/snapshot/1dayago/bucket s3:/some/bucket
Re:Announcing RDS/Aurora Snapshot Export to S3 by sandGorgon on Saturday January 25, 2020 @01:59AM (#22145152)
AFAIK rsync.net doesnt work with RDS (which is a managed solution with no ssh access).
Re:Announcing RDS/Aurora Snapshot Export to S3 by rsync on Saturday January 25, 2020 @01:09PM (#22147600)
I can't speak to RDS specifically, but if you can get RDS into either S3 or Glacier, then there is an easy path to rsync.net: rclone.

The ssh access is not to Amazon, or S3 or RDS, etc. - you run the rclone command at rsync.net, over ssh:

  ssh [email protected] rclone s3:/some/bucket local/rsync.net/dir
Re:Announcing RDS/Aurora Snapshot Export to S3 by Alex3917 on Friday January 24, 2020 @01:48PM (#22141232)
> We use snapshots for regular backups, and have implemented a solutions to share and copy snapshots to a different aws account.

How do you do this if the db is encrypted? Or rather, how do you restore from the encrypted backup in the second account if the master key in the original account is somehow deleted without anyone noticing and intervening before the deletion becomes finalized?

Re:Announcing RDS/Aurora Snapshot Export to S3 by aquark on Friday January 24, 2020 @02:09PM (#22141405)
Good point ... we don't have this db encrypted as the DBMS level. Internally sensitive information is encrypted using keys not managed by KMS
Re:Announcing RDS/Aurora Snapshot Export to S3 by aeyes on Friday January 24, 2020 @12:42PM (#22140559)
The export does type conversion so this is one-way by definition :(.

https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_...

Re:Announcing RDS/Aurora Snapshot Export to S3 by joncrane on Friday January 24, 2020 @01:16PM (#22140934)
The canonical AWS answer for things that you want in Glacier that only support export to S3 is to put a lifecycle policy on the bucket to move to Glacier after 24 hours.

One day in S3 = 10 days in Glacier and since Glacier is presumably for years, the cost difference is trivial.

Re:Announcing RDS/Aurora Snapshot Export to S3 by dpedu on Friday January 24, 2020 @01:27PM (#22141039)
It is not designed to be a backup mechanism.
Re:Announcing RDS/Aurora Snapshot Export to S3 by etaioinshrdlu on Friday January 24, 2020 @02:19PM (#22141513)
I use rds Aurora but i run a daily cron job to export the entire thing with mysqldump + gzip to s3.

I pass some flags to mysqldump to avoid locking the db and otherwise interfering with production. It also dumps from the reader node not the writer.

I also clear some tables and restore it daily to a development DB.

Sadly this looks like it only supports parquet, a rather usual db dump format. Even if, I'm sure, it's way more efficient to process using some modern tools than a text sql dump.

I just like open formats and interoperability and AWS's rds offerings are always just slightly 'off'.

Re:Announcing RDS/Aurora Snapshot Export to S3 by sudhirj on Friday January 24, 2020 @03:13PM (#22141984)
Parquet is an open format. It’s part of the Apache foundation.

Would have preferred CSV as well, though, easier to work with. There’s also a little known feature called S3 Select that allows you to run SQL on S3 files and selectively retrieve data, which is what Parquet is for in the first place.

Re:Announcing RDS/Aurora Snapshot Export to S3 by etaioinshrdlu on Friday January 24, 2020 @05:05PM (#22142900)
Can you load that dump using any open source tools outside of AWS and recreate the database identically? Because you can do that with mysqldump.
Re:Announcing RDS/Aurora Snapshot Export to S3 by derision on Friday January 24, 2020 @07:00PM (#22143529)
I don't think this is meant for recreating the database, but rather exporting large sets for analysis somewhere else. But as far as tools yes, parquet can be parsed by any tool that knows the format. It's not too uncommon
Re:Announcing RDS/Aurora Snapshot Export to S3 by nitesh_aws on Friday January 24, 2020 @11:49AM (#22139933)
Full disclosure - I work for AWS and my opinions are my own
Re:Announcing RDS/Aurora Snapshot Export to S3 by saurik on Friday January 24, 2020 @12:31PM (#22140444)
For those of us who are interested, where might we find some of your hottest opinions?
Re:Announcing RDS/Aurora Snapshot Export to S3 by mabbo on Friday January 24, 2020 @01:04PM (#22140808)
Stealing jeffbarr/jbarr@'s thunder? :D
Re:Announcing RDS/Aurora Snapshot Export to S3 by samokhvalov on Friday January 24, 2020 @06:07PM (#22143233)
Great news. But unfortunately, it is not the original data snapshot as the title says, it's Apache Parquet format. So this is useful for analytical tasks only.

For operational tasks, still, the only option (besides RDS cloning) for Postgres is pg_dump or pg_transport, losing physical layout.

Re:Announcing RDS/Aurora Snapshot Export to S3 by scrollaway on Friday January 24, 2020 @12:58PM (#22140741)
Are there lightweight solutions to reading parquet files in Python? Any time I want to deal with parquet in AWS lambda I have to deal with the entire Pandas suite which is a pain on lambda.
Re:Announcing RDS/Aurora Snapshot Export to S3 by cavisne on Friday January 24, 2020 @01:14PM (#22140922)
One interesting way is to use S3 Select which can read parquet, then you just need a dependency on the AWS sdk
Re:Announcing RDS/Aurora Snapshot Export to S3 by nishantvyas on Friday January 24, 2020 @01:21PM (#22140975)
What happens to bandwidth saturation on the backup database/RDS host? is it unlimited? Capped? user-defined? i.e. would backup impacts the ongoing transactions/packet transfer?
Re:Announcing RDS/Aurora Snapshot Export to S3 by Tobani on Friday January 24, 2020 @01:28PM (#22141051)
snapshots would happen as normal in RDS. This is just the process of moving them "offsite" to s3.
Re:Announcing RDS/Aurora Snapshot Export to S3 by nishantvyas on Friday January 24, 2020 @01:36PM (#22141112)
got it. thanks.
Re:Announcing RDS/Aurora Snapshot Export to S3 by giseir on Friday January 24, 2020 @01:24PM (#22141009)
Excuse me for technical illiteracy, but would that make sense to export these snapshots daily? Is there any way to consistently get the (every 24h) data from the database in S3?
Re:Announcing RDS/Aurora Snapshot Export to S3 by giseir on Friday January 24, 2020 @01:36PM (#22141114)
Let’s say I have a database with 1TB of data. I export it daily to S3 with this Snapshot export. Does it mean I will be adding 1TB every day to my S3 storage?
Re:Announcing RDS/Aurora Snapshot Export to S3 by HatchedLake721 on Friday January 24, 2020 @01:38PM (#22141130)
Yes
Re:Announcing RDS/Aurora Snapshot Export to S3 by giseir on Friday January 24, 2020 @01:39PM (#22141149)
Thanks, now I understand it. Then it makes sense to delete the earlier snapshot and only keep the latest one to not store redundant data.
Re:Announcing RDS/Aurora Snapshot Export to S3 by jungturk on Friday January 24, 2020 @03:03PM (#22141895)
You can also use the CDC features of AWS Data Migration Service to just get incrementals (also in parquet if preferred) rather than full snapshots.

https://aws.amazon.com/blogs/database/aws-dms-now-supports-n...

Re:Announcing RDS/Aurora Snapshot Export to S3 by johnrob on Friday January 24, 2020 @01:20PM (#22140972)
Quick question (answer might be in docs somewhere): can you choose which tables to export?
Re:Announcing RDS/Aurora Snapshot Export to S3 by nitesh_aws on Friday January 24, 2020 @01:22PM (#22140987)
Yes, you can filter. More instruction in the docs.
Re:Announcing RDS/Aurora Snapshot Export to S3 by cmclaughlin on Friday January 24, 2020 @01:53PM (#22141278)
Paying for the full snapshot and not just the filtered data is a bummer
Re:Announcing RDS/Aurora Snapshot Export to S3 by jbverschoor on Friday January 24, 2020 @12:52PM (#22140669)
Nice! More database dumps exposed.
Re:Announcing RDS/Aurora Snapshot Export to S3 by bdcravens on Friday January 24, 2020 @01:54PM (#22141281)
I imagine a significant number of companies rolled their own solutions to dump to s3 already, so this is no less secure.