News for hackers. Stuff that matters.
Re:Classic Data science pipelines built with LLMs by miningape on Sunday February 09, 2025 @08:13AM (#42990464)
"lemme just fire up a dbt workflow to analyse this CSV file"
Re:Classic Data science pipelines built with LLMs by tesch1 on Sunday February 09, 2025 @11:16AM (#42991555)
You may have meant that sarcastically, but i just did that for 2 csv files that i needed to do a bunch of cleanups and joins to analyze. With llm help the whole adventure was easy.
Re:Classic Data science pipelines built with LLMs by miningape on Sunday February 09, 2025 @03:16PM (#42993477)
What I really like to do for this is loading it into SQLite, there are built in macros for reading/writing CSV files. And they're queryable with SQL which makes for a great jumping point to do some basic cleaning, joining and analysis.

This also I'd argue makes the job easier with LLMs since you can ask it to write a SQL query which you can validate / reason about rather than relying on it for transforming the data itself (which I've seen a lot under this post)

Re:Classic Data science pipelines built with LLMs by CalRobert on Sunday February 09, 2025 @03:40PM (#42993715)
Honestly I spend ten times as much effort figuring out people's sloppy notebooks or pandas stuff than when they just use DBT and SQL. And 90% of the time SQL is all they needed.