News for hackers. Stuff that matters.
Re:ML Experiments Management with Git by vinni2 on Thursday November 02, 2023 @06:11PM (#38120997)
I gave up on dvc and instead switched to huggingface and wandb because of the way it handled large files and large local cache it downloaded.
Re:ML Experiments Management with Git by icyfox on Thursday November 02, 2023 @06:20PM (#38121125)
I have a similar harness going for my recent experiments, except instead of hosting with huggingface I have a dataframe with pointers to the files on S3 and then just download them during local preprocessing.

Every time I see DVC mentioned I always feel like the idea was so close (and perhaps right in intuition to use git for everything) but the execution had just enough friction that I looked elsewhere. Small DX improvements really do cascade pretty far.

Re:ML Experiments Management with Git by nerdponx on Thursday November 02, 2023 @06:35PM (#38121317)
[delayed]
Re:ML Experiments Management with Git by unsynced on Thursday November 02, 2023 @06:50PM (#38121495)
FYI, you can use git worktrees [1] to work on multiple branches simultaneously

[1] https://git-scm.com/docs/git-worktree

Re:ML Experiments Management with Git by nerdponx on Thursday November 02, 2023 @08:31PM (#38122593)
Yeah, I know and love that feature for software projects, especially if I need to switch over to a bugfix while I'm deep in a topic branch.

But for a data project it would be a big pain to have separate worktrees just to work around what IMO is a usage anti-pattern to begin with!

Re:ML Experiments Management with Git by shcheklein on Thursday November 02, 2023 @10:51PM (#38123851)
DVC has `dvc exp` that doesn't require creating commits or branches. It's utilizing git custom references (technical details [1]). And it can be visualized in CLI or VS Code.

[1] https://iterative.ai/blog/experiment-refs

[2] https://marketplace.visualstudio.com/items?itemName=Iterativ...

Re:ML Experiments Management with Git by nerdponx on Friday November 03, 2023 @05:35PM (#38135289)
[delayed]
Re:ML Experiments Management with Git by a_bonobo on Friday November 03, 2023 @02:53AM (#38125370)
I've really liked the idea of scidataflow in this context: https://github.com/vsbuffalo/scidataflow

It's neat for research as it stores the data on scientific data repositories like Zenodo and you get DOIs.

Re:ML Experiments Management with Git by esafak on Thursday November 02, 2023 @07:43PM (#38122009)
I haven't used it but git seems like the wrong framework for managing experiments. Even programmers don't especially like git, forget about data scientists. You can't meaningfully version control the data; it's usually binary, and in an object store. The code already lives in a git repository. So what does that leave, the metadata around the experiment?

A relational database like Dolt seems like a better fit. You want to be able to query by experiment name, date, test results, and other metadata.

Let me know if I'm doing it wrong! What's your use case?

Re:ML Experiments Management with Git by IanCal on Friday November 03, 2023 @04:46AM (#38126058)
DVC stores metadata files in your git repo and the actual files somewhere else (local, S3, etc). It handles swapping around the files so they match what you've checked out. It does a lot more but that's the relevant part for your main question.
Re:ML Experiments Management with Git by skadamat on Friday November 03, 2023 @10:59AM (#38129805)
> You can't meaningfully version control the data; it's usually binary, and in an object store. The code already lives in a git repository. So what does that leave, the metadata around the experiment?

Out of the box you can't. We're taking a different approach at work (XetHub). GitHub sees pointer files but we embed rendered views and diffs (supporting more file types incrementally) using a Github app from our service.

Re:ML Experiments Management with Git by ishabytes on Thursday November 02, 2023 @07:45PM (#38122029)
Pachyderm is another alternative. Here's a couple articles that compare the two:

https://www.pachyderm.com/blog/data-versioning-comparing-dvc... https://www.dolthub.com/blog/2022-04-27-data-version-control...

As far as I know, DVC is better than Pachyderm for small datasets, but Pachyderm scales way better

Re:ML Experiments Management with Git by claytonjy on Friday November 03, 2023 @10:41AM (#38129530)
I built an ML pipeline in pachyderm years ago, maybe 2016, and loved using it then. But my data was small-to-medium, and I stuck with CSVs because back then you couldn't get many of the benefits with a binary format.

I think it's a shame it took Airflow and similar several more years to realize "each step is a docker container" is the right way to build a dag. It's not clear to me why Pachyderm was left behind while Prefect and Dagster became serious contenders, and Airflow/Astronomer started recommending everyone use it just like Pachyderm (container per step).

Re:ML Experiments Management with Git by mriet on Thursday November 02, 2023 @07:58PM (#38122152)
Exactly! Ironically, dvc doesn't really scale well and is limited to small-ish files.
Re:ML Experiments Management with Git by shcheklein on Thursday November 02, 2023 @10:48PM (#38123835)
hey, sorry to hear that. Could you share more details please? Were you using specifically for data management?
Re:ML Experiments Management with Git by tsroe on Friday November 03, 2023 @04:54AM (#38126117)
How do you track code-changes between model iterations with this setup though? From looking at wandb, it seems like it does something similar to MLFlow, so it only logs meta-parameters right?
Re:ML Experiments Management with Git by vinni2 on Friday November 03, 2023 @03:00PM (#38133398)
It can log anything you want model config and evaluation metrics along with git commit hash of the code that was used. You can also put pointers to datasets. Easy to compare experiments and choose a model or repeat experiments.
Re:ML Experiments Management with Git by claytonjy on Friday November 03, 2023 @10:34AM (#38129420)
which things do you do with HF, and which with WandB?

HF seems like "GitHub for models and datasets", it has a cool brand and everyone in ML uses it in some capacity. But when it comes to _company_ needs like private datasets/models, experiment tracking, CI integrations, etc. it seems WandB is a superset of HF.

HF has an enterprise offering, but it seems to be de-prioritized, and I think you'd still need WandB or MLFlow for experiment tracking?