News for hackers. Stuff that matters.
Re:Unpaywall: An open database of 31,903,705 free scholarly ... by perihelions on Friday May 06, 2022 @01:51AM (#31281737)
- "To complete the download of the full dataset, please fill out this form, which helps us report usage back to our funders, and we'll immediately provide you with a download link."

https://unpaywall.org/products/snapshot

Is that dataset different from this un-gated one? They're both indexes of Crossref DOI's, and they're both 120 million records long.

https://www.crossref.org/blog/new-public-data-file-120-milli...

Re:Unpaywall: An open database of 31,903,705 free scholarly ... by bnewbold on Friday May 06, 2022 @01:40PM (#31287868)
Unpaywall is based on Crossref DOIs (one-to-one records), and adds information about publicly accessible versions of each work. In theory publishers can register metadata about whether articles are OA or not with Crossref, but the quality and coverage of this metadata is poor in general.

Unpaywall will both check if articles are actually available from the publisher (by following the DOI and parsing the landing page), and by looking for other versions elsewhere on the web (eg, a pre-print). It is simple in theory, but doing this reliably for millions of DOIs from thousands of publishers is a lot of work!

Re:Unpaywall: An open database of 31,903,705 free scholarly ... by IanCal on Friday May 06, 2022 @10:28AM (#31285518)
Unpaywall is different to crossref. The size will be similar as a lot of scientific DOIs are from crossref, and they're a common dataset to base initial data on.

https://unpaywall.org/data-format is the format for unpaywall

https://api.crossref.org/swagger-ui/index.html for crossref info

Re:Unpaywall: An open database of 31,903,705 free scholarly ... by snarkerson on Friday May 06, 2022 @11:27AM (#31286297)
Would not be surprised if the Crossref DB was their starting point.
Re:Unpaywall: An open database of 31,903,705 free scholarly ... by hpiwowar on Monday May 09, 2022 @12:46AM (#31310732)
It is indeed :) (COI: Unpaywall cofounder and dev)