News for hackers. Stuff that matters.

A comment on A Billion NYC Taxi and Uber Rides in AWS Redshift. Parent comment

Re:A Billion NYC Taxi and Uber Rides in AWS Redshift by JimmyAustin on Sunday February 14, 2016 @03:57AM (#11097634)
From my experience analysing similar sized datasets, it really comes down to the structure of the query and dataset. If you nail your dist keys and your queries don't have many large joins, it's incredibly quick. If your dist keys aren't great and you have many joins, it will take a long long time.
[ Parent | on HN ]
Re:A Billion NYC Taxi and Uber Rides in AWS Redshift by stingraycharles on Sunday February 14, 2016 @04:12AM (#11097657)
Exactly. You have to keep in mind that other than distkeys / sortkeys, there are no indexes. So all you can do to speed things up is organize your data better.

For a ballpark figure, we work with a ~ 500million rows / 12 columns data set, and complex queries with a lot of joins typically take around 10 - 20 minutes to complete on Redshift's smallest cluster ($250/m).

[ Parent | on HN ]