News for hackers. Stuff that matters.

A comment on DBT Founder: Is the "Modern Data Stack" Still a Useful Idea?. Parent comment

Re:DBT Founder: Is the "Modern Data Stack" Still a Useful Idea? by tomnipotent on Sunday February 11, 2024 @08:44PM (#39340569)
> allows me to run dplyr operations on database tables

Which require that data take a round trip between the R process and the database. It's not uncommon for these jobs to spend more time in the read/write step than doing meaningful work, and why I prefer dbt when possible to keep transformations as close to the data as possible.

[ Parent | on HN ]
Re:DBT Founder: Is the "Modern Data Stack" Still a Useful Idea? by nograpes on Sunday February 11, 2024 @10:27PM (#39341025)
The second point the grandparent made was that they were using dbplyr, which allows you to avoid having the data take a round trip between the R process and the database.
[ Parent | on HN ]
Re:DBT Founder: Is the "Modern Data Stack" Still a Useful Idea? by leledavid on Wednesday February 14, 2024 @02:53PM (#39374384)
The dbplyr package takes your R code and executes it in your database, returning a remote dataframe. This is quite useful because there is no mvoving data back and forth.
[ Parent | on HN ]