Redshift Spectrum – Exabyte-Scale In-Place Queries of S3 Data

Re:Redshift Spectrum – Exabyte-Scale In-Place Queries of S3 ... by teej on Wednesday April 19, 2017 @10:04PM (#14153402)

Just for comparison's sake, I recreated Jeff's example queries on my in-house table that's ~8B rows. COUNT(*) takes 6s and the SELECT with 2 groups, 2 sums, and 2 averages takes 52s. The downside is that the table takes 600GB of disk space compressed which is a major chunk of our cluster. But here's the kicker - I would never query this table for data I needed in under 5 min so my current performance is meaningless.

Given our access patterns for this table, I'm going to investigate using Redshift Spectrum for it. It seems like a huge win for us.

[ Parent | on HN ]

Re:Redshift Spectrum – Exabyte-Scale In-Place Queries of S3 ... by andrioni on Thursday April 20, 2017 @06:28AM (#14155466)

>The downside is that the table takes 600GB of disk space compressed which is a major chunk of our cluster.

And that it probably takes one hour or more to fully reload it if you ever need to.

[ Parent | on HN ]

Re:Redshift Spectrum – Exabyte-Scale In-Place Queries of S3 ... by jnordwick on Wednesday April 19, 2017 @11:54PM (#14153813)

What are you using?

[ Parent | on HN ]

Re:Redshift Spectrum – Exabyte-Scale In-Place Queries of S3 ... by teej on Thursday April 20, 2017 @12:06AM (#14153864)

Sorry, this is just vanilla redshift.

[ Parent | on HN ]