News for hackers. Stuff that matters.
Re:DBT Founder: Is the "Modern Data Stack" Still a Useful Idea? by fifilura on Monday February 12, 2024 @06:54AM (#39343863)
I am curious, what do you perceive as the difference between data transformations and "wrangling raw data into tables"?

Edit: I am also curious why you consider SQL to be inferior for the latter?

Re:DBT Founder: Is the "Modern Data Stack" Still a Useful Idea? by jochem9 on Wednesday February 14, 2024 @01:32AM (#39367133)
The difference is that raw data can come in many shapes (e.g. impossibly nested jsons) and unexpected quality (changing field names...). I cannot easily work with this in SQL and definitely not write tests to cover the ever increasing complexity of dealing with messy data.

After that step the data is a lot more uniform, so then it's easy to use SQL.