Likewise, we curate a list of large open source datasets here -> https://datasets.activeloop.ai/docs/ml/, but our main thing isn't aggregating datasets (focus for HF datasets), but rather providing people with a way to manage their data efficiently. That being said, all of the 125+ public datasets we have are available in seconds with one line of code. :)
We haven't benchmarked against HF datasets in a while, but Deep Lake's dataloader is much, much faster in third-party benchmarks (see this https://arxiv.org/pdf/2209.13705 and here for an older version, that was much slower than what we have now, see this: https://pasteboard.co/la3DmCUR2iFb.png). HF under the hood uses Git-LFS (to the best of my knowledge) and is not opinionated on formats, so LAION just dumps Parquet files on their storage.
While your setup would work for a few TBs, scaling to PB would be tricky including maintaining your own infrastructure. And yep, as you said NAS/NFS would neither be able to handle the scale (especially writes with 1k workers). I am also slightly curious about your use of mmap files with image/video compressed data (as zero-copy won’t happen) unless you decompress inside the GPU ;), but would love to learn more from you! Re: pricing thanks for the feedback, storage is one component and customly priced for PB-scale workloads.
https://www.activeloop.ai/pricing/
which says
"Deep Lake Enterprise" and then "10TB of managed data (total)".
To me, that read as if the Enterprise plan is limited to a maximum of 10TB.
If you find any other points of confusion, please send them our way, and we will fix it, the community has been instrumental over the years in iterating on the product! :)
You can then visualize your datasets if their stored on our cloud, in AWS/GCP, or you can drag and drop your local dataset in Deep Lake format into our UI (https://docs.activeloop.ai/dataset-visualization)
We do, with version control, Python based dataloader and dataset format being open source! Please check out https://github.com/activeloopai/deeplake.
And they also have ready-to-use scripts for A LOT of the usual datasets: https://huggingface.co/datasets
including LAION 400M and LAION 2B: https://huggingface.co/datasets/laion/laion2B-en
TL;DR DagsHub lets you stream datasets from any repo you can access for free. We have open-source datasets for various tasks and domains (image, video, MRI, audio, etc.) that you can use, or upload yours and stream it. Learn more - https://dagshub.com/docs/feature_guide/direct_data_access/
How does it work? Every DagsHub repo comes with a configured remote storage, where users can host models, datasets, or any other large file.
We recently added a new capability to our open-source client and free-to-use API that enables the streaming of files stored on DagsHub Storage.
It enables access to any dataset stored on DagsHub, stream it to your machine, version it, and upload it to your DagsHub repo - all from your python code.
I think the coolest part of this feature is that it doesn't require any modifications to your code base or data format.
You can find more info about it here -> https://dagshub.com/docs/feature_guide/direct_data_access/
Feel free to reach out if you have any other questions (:
In my opinion, this space will be won based on price per throughput.
But you're charging $1k monthly for 1TB? And my hobbyist speech recognition projects (30TB) vastly exceed your enterprise plan? That discrepancy makes it challenging to believe that your team has as much experience with actual AI projects, as the reference to self-driving cars would suggest.
Anything that easily fits onto a consumer NAS is probably not an enterprise data lake. A one-time €1700 purchase (equivalent to 2 months of your service) will buy me a TS-464-4G with 32TB RAID1 and 5 years of warranty.