awgupta

This is a different feature than PiTR. We have both. They serve different needs.

that's correct.

That is a different feature (although a cool one). They provide the ability to run a query as of a point in time. We are moving the database backward in time (which matters for running applications)

Also, generally DDL statements like DROP TABLE typically autocommit.

You can also clone the database volume in Aurora and then just backtrack one of the volumes. That should help you ensure you have a version available for forensic analysis.

You're right - it is a lot faster. That's because we don't change any data blocks. we just mark a portion of the log to be ignored.

That's not correct. We made a change to mark portions of our log-structured storage as though they should not have occurred. It is a totally different approach than point-in-time restore.

As the previous commentor said, this isn't point in time restore or a change in our processes. It is a change to our database and storage engine to mark regions of log-structured storage as if they did not occur. That's why we can complete a Backtrack in a few seconds - we're just moving some pointers around and restarting the database.

We've had point in time restore for quite some time. Backtrack is different. It moves you to a different point using the same instance. Since we don't do destructive writes to blocks (it is log-structured storage), we can simply mark a portion of the log as "ignored". It is a server feature, not a UI enhancement.

no, we don't perform destructive writes so we are able to simply mark a region of the log as to be skipped. this approach also allows us to move back and forth within the time domain even after a backtrack.

that's correct.

that said, I agree that approximate functions should be part of a modern database system. Redshift has approximate count distinct (based on hyperloglog) and approximate percentiles (based on quantile summaries)

The constraint for Redshift Spectrum queries tends to be between S3 and Spectrum. The graphs Jeff shows are in the Redshift cluster itself which, for the sample query he is running, is lightly loaded. We are working on upgrading our performance dashboards...

it really depends on what you are doing. A large data set shouldn't be limited to longitudinal analysis. If you're storing every log record or every stock bid/ask, there may be times that you need to understand the specifics of what exactly was going on. There may be a lot of filtering on the underlying corpus for these sorts of exact match queries, but data set sizes continue to grow.

The underlying tech stacks are totally different.

RDS just cleared HIPAA and BAA. See here: http://aws.amazon.com/compliance/hipaa-compliance/

Working on Aurora...

You can think of Aurora as a single-instance database where the lower quarter is pushed down into a multi-tenant scale-out storage system. Transactions, locking, LSN generation, etc all happen at the database node. We push log records down to the storage tier and Aurora storage takes responsibility for generation of data blocks from logs.

So, the ACI components of ACID are all done at the database tier using (largely) traditional techniques. Durability is where we're using distributed systems techniques around quorums, membership management, leases, etc, with the important caveat that we have a head node generating LSNs, providing a monotonic logical clock, and avoiding those headaches.

Our physical read replicas receive redo log records, update cached entries and have readonly access to the underlying storage tier. The underlying storage is log-structured with nondestructive writes, so we can access data blocks in the past of what is current at the write master node - that's required if the replica needs to read a slightly older version of a data block for consistency reasons.

Make sense?

We've made a number of improvements relative to MySQL, for example with large numbers of tables and the results set cache. There are some improvements on large tables and schema changes, but quite a bit more to be done in those areas. You can contact our PM team at aurora[dash]pm[at]amazon[dot]com to give us the issues that matter most. Feedback is how we prioritize!

That is the basic idea on such things, though distributed protocols become complex when you have to consider split-brain, multiple concurrent failures, etc which all can occur in large-scale events.

Aurora leverages Amazon RDS Multi-AZ in this area. Their implementation has been battle-tested over many years and many many db instances.

it is a per-instance number. For those SysBench numbers, we are running an r3.8xlarge instance type.

Aurora isn't implemented based on either Galera or NDB.

For TPC-C like benchmarks, you can run: 1) CloudHarmony: https://github.com/cloudharmony/oltpbench 2) Percona: https://code.launchpad.net/~perconadev/perconatools/tpcc-mys...

We've found it easier to load large datasets using CloudHarmony but have run both.

I'd recommend reading through http://d0.awsstatic.com/product-marketing/Aurora/RDS_Aurora_.... It is oriented towards SysBench, but will help you set up your clients to have enough network throughput to run a full test.

Generally, we find the performance comparison improves on large instances, high throughput workloads, or when the data set does not fit in RAM.