Re:Everyone wants to do the model work, not the data work [pdf]
by teruakohatu on Monday March 29, 2021 @03:51AM
(#26618685)
Thanks for the information. So the 54 gigabytes is just plain text JSON? How big is the full image dataset?
Re:Everyone wants to do the model work, not the data work [pdf]
by sillysaurusx on Tuesday March 30, 2021 @12:19AM
(#26631226)
Good question. It's 100m images, but I haven't tried to find how large all of it is.
We downloaded a subset of 3 million images, which is apparently 379.77 GiB. So a linear extrapolation would be (100/3*379.77) = ~12,659 GiB for the full 100m images.
12TB really isn't too bad. It's massive, yes, but imagenet 21k is 1.2TB.
Here's curl -fsSL https://battle.shawwn.com/sdc/f100m/yfcc100m_dataset.json | head -n 1, which should give you all the info about the structure. Or most of it.
I just dump those to disk and then use `jq` to extract the item_download_url.FWIW, the full ceiling search finished with 9,325 results. Here's the first 1k.
https://cdn.gather.town/storage.googleapis.com/gather-town.a...