News for hackers. Stuff that matters.
Re:Starting July 1, Academic Publishers Can't Paywall NIH-Fu... by sadiq on Thursday May 01, 2025 @12:32PM (#43859978)
This is good though it's not clear whether these papers will appear in the PMC Open Access subset (https://pmc.ncbi.nlm.nih.gov/tools/openftlist/) and be bulk downloadable.

I've been doing some work with colleagues at Cambridge and Imperial over the last year on using LLMs to improve evidence synthesis, primarily trying to find papers on the effectiveness of certain Conservation interventions. It's becoming clear that you really need to move beyond screening papers only by title and abstract - there's often information buried deep within papers that can only be found with access to full text. My colleague Anil Madhavapeddy has written a bit about our adventures in trying to ingest full-text academic papers: https://anil.recoil.org/notes/uk-national-data-lib

Re:Starting July 1, Academic Publishers Can't Paywall NIH-Fu... by shishy on Thursday May 01, 2025 @03:44PM (#43862469)
Yes, it depends on what you're doing; for general paper discovery / search tasks, title abstract can be enough (which is also why Springer and Elsevier have been pulling even their abstracts from sources like OpenAlex).

But for something like that you need full texts to look into results sections. I'm very curious how you're dealing with information contained in tables, or if you're dealing with snippets of text from the full-text alone. Have you poked around Elicit yet?

Re:Starting July 1, Academic Publishers Can't Paywall NIH-Fu... by a_bonobo on Thursday May 01, 2025 @09:21PM (#43865217)
I've recently had this problem where the important information (number of study participants, and how many were filtered out during which step) were only encoded in figures, not in the text. Maddening.
Re:Starting July 1, Academic Publishers Can't Paywall NIH-Fu... by spookie on Thursday May 01, 2025 @01:19PM (#43860602)
Do you know of any ready to use alternatives to title and abstract screening? Wondering about it since I'm in the weeds of doing so.
Re:Starting July 1, Academic Publishers Can't Paywall NIH-Fu... by tough on Thursday May 01, 2025 @01:54PM (#43861092)
what do you mean exactly? I was suprised how with grobid many of at least the arXiv papers are easily converted to xml for better processing than PDF.

Most of the papers are constructed from their latex sources so there's an easy way to undo it i guess.

https://github.com/kermitt2/grobid

Re:Starting July 1, Academic Publishers Can't Paywall NIH-Fu... by shishy on Thursday May 01, 2025 @03:42PM (#43862440)
grobid is a wonderful resource, patrice did an awesome job (I used it at my previous job at scite.ai)
Re:Starting July 1, Academic Publishers Can't Paywall NIH-Fu... by spookie on Thursday May 01, 2025 @11:19PM (#43865867)
that's exactly what I needed!
Re:Starting July 1, Academic Publishers Can't Paywall NIH-Fu... by tough on Friday May 02, 2025 @05:23AM (#43867718)
glad to hear!