News for hackers. Stuff that matters.
Re:Classifying all of the pdfs on the internet by tokai on Monday August 19, 2024 @09:55AM (#41291083)
Yeah 8TB is really tiny. Google scholar was estimated to index 160.000.000 pdfs in 2015.[0] If we assume that a third of those are not behind paywalls, and average pdf size is 1mb, its ends up as something above 50TB of documents. Almost ten years later the number of available pdfs of just scholarly communication should be substantially higher.

[0] https://link.springer.com/article/10.1007/s11192-015-1614-6

Re:Classifying all of the pdfs on the internet by elorant on Monday August 19, 2024 @01:57PM (#41293159)
Anna's archive has some 300M pdfs.
Re:Classifying all of the pdfs on the internet by tokai on Monday August 19, 2024 @02:11PM (#41293313)
We're talking about the open web here. But yeah that's the point, the dataset is unreasonably small.