Re:Data Version Control
by causal on Saturday October 19, 2024 @06:17PM
(#41891218)
This useful for large binaries?
[ Parent
| on HN ]
Yes. And if you track transformations of the binaries or ml training
Would appreciate a good answer to this question. I deal with large medical imaging data (DICOM) and i cannot tell whether it's worth it and/or feasible.
Re:Data Version Control
by thangngoc89 on Saturday October 19, 2024 @10:46PM
(#41892510)
It's very much feasible. I'm currently using DVC for DICOM, the repo has growth to about 5TB of small dcm files (less than < 100KB each). We use a NFS mounted NAS for development but the DVC's cache needs to be on the NVMe, otherwise performance would be terrible.
Re:Data Version Control
by tomnicholas1 on Sunday October 20, 2024 @09:11AM
(#41895148)
You should look at Icechunk. Your imaging data is structured (it's a multidimensional array), so it should be possible be to represent it as "Virtual Zarr". Then you could commit it to an Icechunk store.
Re:Data Version Control
by mkbehbehani on Saturday October 19, 2024 @09:17PM
(#41892136)
Yes, I’ve been using it for about a year to populate databases with a reference DB dump. The current file is about 18GB. I use cloudflare R2 as the backing store so even though it’s being pulled very frequently the cloudflare bill is a few bucks per month.