DataChain: DBT for Unstructured Data

Re:DataChain: DBT for Unstructured Data by mpeg on Monday November 04, 2024 @01:10PM (#42044366)

It took me a minute to grok what this was for, but I think I like it

It doesn't really replace any of the tooling we use to wrangle data at scale (like prefect or dagster or temporal) but as a local library it seems to be excellent, I think what confused me most was the comparison to dbt.

I like the from_* utils and the magic of the Column class operator overloading and how chains can be used as datasets. Love how easy checkpointing is too. Will give it a go

[ Parent | on HN ]

Re:DataChain: DBT for Unstructured Data by dmpetrov on Monday November 04, 2024 @01:24PM (#42044515)

Yes, it's not meant to replace data engineering tools like Prefect or Temporal. Instead, it serves as a transformation engine and ad-hoc analytics for images/video/text data. It's pretty much DBT use case for text and images in S3/GCS, though every analogy has its limits.

Try it out - looking forward to your feedback!

[ Parent | on HN ]