News for hackers. Stuff that matters.
Re:What Is Dbt and Why Are Companies Using It? by ramraj07 on Thursday December 02, 2021 @10:36PM (#29426366)
The issue with dbt is now folks end up needing to spin up the entire repository to run a bunch of sql queries. The issue is this is a tool that’s targeted at teams that don’t have rigorous engineering backgrounds so it’s bound to be used in suboptimal ways. If the dbt team can do things about it, it might be even better.
Re:What Is Dbt and Why Are Companies Using It? by adeelk93 on Friday December 03, 2021 @12:12AM (#29426837)
Before dbt, I didn't even really think about structuring how I'd be running my queries. My experience with it has been - it's made it clear to me what has been sub-optimally laid out. Organize my fact tables by subject, adopt a common schema before applying business logic, utilize re-usable views instead of copy-pasting CTEs, etc.

And I like its tight git integration. The alternative, quite often, is a folder of scripts on an analyst's desktop.

Re:What Is Dbt and Why Are Companies Using It? by CactusOnFire on Friday December 03, 2021 @12:14AM (#29426845)
Yeah "forcing version control" seems like a feature (pertaining to data governance), not a bug.
Re:What Is Dbt and Why Are Companies Using It? by bearfood on Friday December 03, 2021 @12:16AM (#29426857)
Shameless, but very relevant plug:

My team have been working to solve this problem (and more). We recently released our CLI tool, palm, as OSS. Along with a plugin, palm-dbt, which we designed specifically for working with dbt projects in an optimal way.

Palm requires running code in containers, so teams can ensure dependencies and versions are correct. Then the plugin provides sensible defaults, with solutions for common problems (like idempotent dev/ci runs) and an abstraction in front of the most common commands you need to run in the container.

It's been a huge boon for our analysts productivity, improved the reliability and quality of our models, and decreased onboarding times for new people joining our data team.

Links: https://github.com/palmetto/palm-cli https://github.com/palmetto/palm-dbt