News for hackers. Stuff that matters.
Re:Everyone wants to do the model work, not the data work [pdf] by teruakohatu on Monday March 29, 2021 @03:51AM (#26618685)
Thanks for the information. So the 54 gigabytes is just plain text JSON? How big is the full image dataset?
Re:Everyone wants to do the model work, not the data work [pdf] by sillysaurusx on Tuesday March 30, 2021 @12:19AM (#26631226)
Good question. It's 100m images, but I haven't tried to find how large all of it is.

We downloaded a subset of 3 million images, which is apparently 379.77 GiB. So a linear extrapolation would be (100/3*379.77) = ~12,659 GiB for the full 100m images.

12TB really isn't too bad. It's massive, yes, but imagenet 21k is 1.2TB.