Ask HN: How do you like to version datasets for production ML systems?

Posted by wskish on Tuesday November 29, 2022 @01:26PM
from the ask-hn dept.

At my previous company (industrial ai vision platform) we had an ad-hoc mechanism for continuously versioning datasets for user applications that we built internally several years ago.

I am curious how folks are versioning datasets now in practice. Platforms like Hugging Face and Weights and Biases seem to provide good abstractions for dataset versioning. Any feedback on the suitability of these for production systems, or other systems, patterns, or best practices that you have found to work well?

( 2 comments | 3 points | on HN )

Re:Ask HN: How do you like to version datasets for productio... by kristenkehrer on Tuesday November 29, 2022 @01:55PM (#33791593)

I use CometML (because I work there), but creating data artifacts and versioning my data have made life so much easier.. especially if I put a project down for a bit and try to pick it back up.

[ Parent | on HN ]

Re:Ask HN: How do you like to version datasets for productio... by wskish on Wednesday November 30, 2022 @02:05AM (#33797811)

Any talking points on how CometlML compares to HF and wandb for this purpose?

[ Parent | on HN ]