I built an ML pipeline in pachyderm years ago, maybe 2016, and loved using it then. But my data was small-to-medium, and I stuck with CSVs because back then you couldn't get many of the benefits with a binary format.
I think it's a shame it took Airflow and similar several more years to realize "each step is a docker container" is the right way to build a dag. It's not clear to me why Pachyderm was left behind while Prefect and Dagster became serious contenders, and Airflow/Astronomer started recommending everyone use it just like Pachyderm (container per step).
https://www.pachyderm.com/blog/data-versioning-comparing-dvc... https://www.dolthub.com/blog/2022-04-27-data-version-control...
As far as I know, DVC is better than Pachyderm for small datasets, but Pachyderm scales way better