News for hackers. Stuff that matters.

A comment on DBT Founder: Is the "Modern Data Stack" Still a Useful Idea?. Parent comment

Re:DBT Founder: Is the "Modern Data Stack" Still a Useful Idea? by jochem9 on Monday February 12, 2024 @01:35AM (#39342129)
Imo dbt sits in the analysts space. SQL is the language they know and dbt is the best solution to scale that.

In that sense it should only be used for last mile data transformations. Not for wrangling raw data into neat tables.

[ Parent | on HN ]
Re:DBT Founder: Is the "Modern Data Stack" Still a Useful Idea? by fifilura on Monday February 12, 2024 @06:54AM (#39343863)
I am curious, what do you perceive as the difference between data transformations and "wrangling raw data into tables"?

Edit: I am also curious why you consider SQL to be inferior for the latter?

[ Parent | on HN ]
Re:DBT Founder: Is the "Modern Data Stack" Still a Useful Idea? by jochem9 on Wednesday February 14, 2024 @01:32AM (#39367133)
The difference is that raw data can come in many shapes (e.g. impossibly nested jsons) and unexpected quality (changing field names...). I cannot easily work with this in SQL and definitely not write tests to cover the ever increasing complexity of dealing with messy data.

After that step the data is a lot more uniform, so then it's easy to use SQL.

[ Parent | on HN ]