Re:Redshift Spectrum – Exabyte-Scale In-Place Queries of S3 ...
by teraflop on Wednesday April 19, 2017 @03:05PM
(#14150735)
Yeah, I think filtering is a big part of it. If you want to answer a statistical question about the entire dataset, then a random sample is probably good enough. If you want to drill down and do an analysis that only looks at a particular narrow slice of the data, then it's likely that the corresponding subset of your sample isn't big enough to be meaningful.
(You can pre-filter or pre-aggregate before sampling, but that assumes you know a priori what types of queries you'll want to do.)
(You can pre-filter or pre-aggregate before sampling, but that assumes you know a priori what types of queries you'll want to do.)