Re:Data Version Control
by natsucks on Saturday October 19, 2024 @08:48PM
(#41892002)
Would appreciate a good answer to this question. I deal with large medical imaging data (DICOM) and i cannot tell whether it's worth it and/or feasible.
[ Parent
| on HN ]
Re:Data Version Control
by thangngoc89 on Saturday October 19, 2024 @10:46PM
(#41892510)
It's very much feasible. I'm currently using DVC for DICOM, the repo has growth to about 5TB of small dcm files (less than < 100KB each). We use a NFS mounted NAS for development but the DVC's cache needs to be on the NVMe, otherwise performance would be terrible.
Re:Data Version Control
by tomnicholas1 on Sunday October 20, 2024 @09:11AM
(#41895148)
You should look at Icechunk. Your imaging data is structured (it's a multidimensional array), so it should be possible be to represent it as "Virtual Zarr". Then you could commit it to an Icechunk store.