News for hackers. Stuff that matters.
Re:ML Experiments Management with Git by ishabytes on Thursday November 02, 2023 @07:45PM (#38122029)
Pachyderm is another alternative. Here's a couple articles that compare the two:

https://www.pachyderm.com/blog/data-versioning-comparing-dvc... https://www.dolthub.com/blog/2022-04-27-data-version-control...

As far as I know, DVC is better than Pachyderm for small datasets, but Pachyderm scales way better

Re:ML Experiments Management with Git by claytonjy on Friday November 03, 2023 @10:41AM (#38129530)
I built an ML pipeline in pachyderm years ago, maybe 2016, and loved using it then. But my data was small-to-medium, and I stuck with CSVs because back then you couldn't get many of the benefits with a binary format.

I think it's a shame it took Airflow and similar several more years to realize "each step is a docker container" is the right way to build a dag. It's not clear to me why Pachyderm was left behind while Prefect and Dagster became serious contenders, and Airflow/Astronomer started recommending everyone use it just like Pachyderm (container per step).