News for hackers. Stuff that matters.

Ask HN: Python package for managing science datasets

I was looking for a Python package to manage science datasets, mainly description, download, extraction, on-disk handling, etc... ML frameworks tend to roll their own:

- https://github.com/nilearn/nilearn/blob/master/nilearn/datasets/utils.py - https://github.com/tensorflow/datasets/blob/master/tensorflow_datasets/core/download/download_manager.py - https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/datasets/base.py - https://github.com/pytorch/vision/blob/master/torchvision/datasets/mnist.py

I was hoping that somebody had made a generic package for this purpose so that I don't roll yet another one.

( 4 comments | 2 points | on HN )
Re:Ask HN: Python package for managing science datasets by amirouche on Tuesday June 04, 2019 @01:29PM (#20097221)
Re:Ask HN: Python package for managing science datasets by kelsolaar on Tuesday June 04, 2019 @09:25PM (#20100746)
That is awesome (pun intended), thanks!
Re:Ask HN: Python package for managing science datasets by swaroop on Tuesday June 04, 2019 @11:31AM (#20095943)
How about https://dvc.org/ ?
Re:Ask HN: Python package for managing science datasets by kelsolaar on Tuesday June 04, 2019 @09:28PM (#20100756)
Interesting, I did not think about this way at all, I will take a look.