Re:1.1B Taxi Rides with 20 Nvidia Tesla P100s and BrytlytDB
by buremba on Wednesday December 13, 2017 @11:02AM
(#15915055)
Real OLAP queries usually don't have JOINs. Instead, the data is denormalized at ingestion nad WINDOW, CTE ans subqueries are used at query time.
Re:1.1B Taxi Rides with 20 Nvidia Tesla P100s and BrytlytDB
by georgewfraser on Wednesday December 13, 2017 @11:09AM
(#15915095)
This was true 10 years ago. Many people building data warehouses today just use a normalized schema and write views with lots of joins. It’s a much simpler workflow and data warehouses have gotten so fast that this actually works.
Re:1.1B Taxi Rides with 20 Nvidia Tesla P100s and BrytlytDB
by buremba on Wednesday December 13, 2017 @12:01PM
(#15915481)
I don't know any technology that makes the JOINs efficient enough for low latency queries. If you're joining two tables that have billions of rows, you need too much memory and CPU resources in order to be able to map them. Do you know any DBMS that focuses on OLAP systems and suggest normalization?
Re:1.1B Taxi Rides with 20 Nvidia Tesla P100s and BrytlytDB
by StreamBright on Wednesday December 13, 2017 @11:46AM
(#15915383)
I can just second that. Most DWHs arr denormalized because columnar storage formats love repetition.
I hear you about compatibility. I think this benchmark would be greatly improved by rewriting the query to include a subquery and a fact-to-fact join using the same table. For example, you could do a subquery where you find the longest and shortest ride for each driver each day, and then join that back to the full table and calculate something. That would do a much better job of exercising the key features of the query planner, while still being simple and compatible.