Re:I'm a programmer – how can I help SciHub?
by andyxor on Monday July 12, 2021 @01:31AM
(#27807354)
that would be cool, the only problem is size of data, about half a petabyte of pdfs uncompressed (77TB compressed). OCR and indexing at this scale would be pretty costly.
[ Parent
| on HN ]
Also papers do not follow any formatting standard, as a human that does not bother us too much, but OCR will have a hard time making sense of multiple columns with random text above and below, figure captions that are separated by the main text with only a little bit of vertical space, etc...
There are some projects that try to automate that, but so far all I have used needed some manual intervention.