Re:Data-Processing Frameworks Benchmark: Redshift, Hive, Sha...
by keithgabryelski on Monday September 30, 2013 @12:32AM
(#6467805)
I also don't see a schema that is tuned for redshift.
There is no description of a sortkey or a distribution key (let alone which compression encoding was used)
And at least for the first query you could use UNLOAD instead of select (depending on how you are managing the data coming out of redshift it might be a reasonable solution and doesn't force all your data through the leader node constrained by whatever client driver you are using to read the results).
Instead of trying to select out of redshift this much data -- select the data into another table or (again) use unload.
That totally confuses columnar compression with columnar I/O, an error I've been railing against for several years, e.g. in http://www.dbms2.com/2011/02/06/columnar-compression-databas... (I.e., ever since Oracle tried to popularize the confusion.) But this is a particularly bad instance.