News for hackers. Stuff that matters.
Re:DBT Founder: Is the "Modern Data Stack" Still a Useful Idea? by fifilura on Monday February 12, 2024 @02:40AM (#39342476)
I am slightly confused by your reply. Can you elaborate a bit, was it good or bad?
Re:DBT Founder: Is the "Modern Data Stack" Still a Useful Idea? by wodenokoto on Monday February 12, 2024 @04:27AM (#39343033)
You can develop a database schema for your application on dummy data, and test your business logic on dummy data. But you can't develop a machine learning model on dummy data.

It is best practice to keep live data out of your development environment for normal software development, but it is impossible for machine learning projects.

Re:DBT Founder: Is the "Modern Data Stack" Still a Useful Idea? by tomrod on Monday February 12, 2024 @08:16AM (#39344405)
Yep. I advise my clients to have a controlled, sandbox partition of prod. Call it the whiteboard or what have you, things can be erased or you can cache model version runs. Your data transforms can happen in dev but training should happen against real data. Synthetic data can be a compromise but that can sharply bias your model and, since it a harder lift, often multiplies data prep time
Re:DBT Founder: Is the "Modern Data Stack" Still a Useful Idea? by fifilura on Monday February 12, 2024 @04:37AM (#39343080)
Yes, this is what I thought and exactly what I am experiencing right now.

It is somewhat frustrating to see the engineering/platform team spending time on this kind of abstraction that is essentially useless for doing the work we need to do.