Re:What Is the Data Lakehouse Pattern?
by rpedela on Wednesday September 15, 2021 @01:30AM
(#28535579)
I think using a data warehouse as your data lake or lake house is optimal. Even for data that isn't relational. Storage is so cheap now and is decoupled from compute costs for several providers that I don't even give it a thought. You get a fast, scalable SQL interface which is still nice and useful for non-relational data. Then all, or most, of the transformations needed for analysis can be pure SQL using a tool like DBT. In my experience, it greatly simplifies the entire pipeline.
[ Parent
| on HN ]
Re:What Is the Data Lakehouse Pattern?
by CRConrad on Wednesday September 15, 2021 @10:10AM
(#28539245)
> pure SQL using a tool like DBT
I don't get it... Looks to me like DBT is a Python SQL wrapper / big library that among other things includes an SQL generator / something else like that -- but not "pure" SQL?
Re:What Is the Data Lakehouse Pattern?
by rpedela on Wednesday September 15, 2021 @01:14PM
(#28541740)
DBT has two main innovations. First, everything is a SELECT statement and DBT handles all the DDL for you. You can handle DDL yourself if you have a special case too. Second, the ref/source macros build a DAG of all your models so you don't have to think about build order. There are other innovations but those are the main ones.
You can give it truly pure SQL in both models and scripts, and mixing in Jinja if you need it for dynamic models. But I'd recommend at least using ref/source.