Re:Everyone wants to do the model work, not the data work [pdf]
by teruakohatu on Monday March 29, 2021 @03:51AM
(#26618685)
Thanks for the information. So the 54 gigabytes is just plain text JSON? How big is the full image dataset?
[ Parent
| on HN ]
Re:Everyone wants to do the model work, not the data work [pdf]
by sillysaurusx on Tuesday March 30, 2021 @12:19AM
(#26631226)
Good question. It's 100m images, but I haven't tried to find how large all of it is.
We downloaded a subset of 3 million images, which is apparently 379.77 GiB. So a linear extrapolation would be (100/3*379.77) = ~12,659 GiB for the full 100m images.
12TB really isn't too bad. It's massive, yes, but imagenet 21k is 1.2TB.