News for hackers. Stuff that matters.
Re:Internet Archive Scholar: Search Millions of Research Papers by bnewbold on Tuesday March 09, 2021 @02:40PM (#26403049)
This service was hinted at back in September, but is now formally announced and live at https://scholar.archive.org

Related previous post: https://news.ycombinator.com/item?id=24485444

Much of the catalog functionality can be accessed from the fatcat.wiki API (https://api.fatcat.wiki/redoc). Scholar adds a search index over the body content of papers, and we are still thinking through how to make this available through a public API without slowing down query latency even more.

Folks here might also be interested in this CLI for interfacing with the catalog and making edits: https://gitlab.com/bnewbold/fatcat-cli

Re:Internet Archive Scholar: Search Millions of Research Papers by breck on Tuesday March 09, 2021 @03:01PM (#26403376)
I absolutely love everything about it (the logo <3).

Super fast. All my test searches returned what I was looking for.

What is your relationship with semantic scholar like?

Any plans to integrate ranking signals like references, etc?

I'm going to double my monthly donation. This is great.

Re:Internet Archive Scholar: Search Millions of Research Papers by bnewbold on Tuesday March 09, 2021 @03:18PM (#26403598)
Thank you for the kind words!

We are friendly with Semantic Scholar, and have used their "open corpus" dumps as one of several URL seed lists for crawling in the past. Their search and discovery tech is more sophisticated than ours is likely to be any time soon (https://medium.com/ai2-blog/building-a-better-search-engine-...). We would love to get to the place where groups like AI2, which are primarily research-oriented, could build on an existing open catalog and corpus, and not need to duplicate time crawling, merging catalogs, cleaning metadata, etc. As of today Microsoft Academic (used by Semantic Scholar) might be a better option.

Want to be thoughtful about ranking signals, and are deeply skeptical of journal impact factor, h-index, and most bibliometrics. "Has this been cited more than a handful of times" seems like a reasonable coarse boost. Hope to include more curated signals, like "won a paper prize", "journal in DOAJ and other reviewed indices", etc.

Have been working on a citation graph, keep an eye out for something about that in coming months. One cool thing we hope to do with the citation graph is find "missing works" not yet in the catalog (eg, don't have a DOI, especially for pre-1990 era).

Re:Internet Archive Scholar: Search Millions of Research Papers by curiouscats on Tuesday March 09, 2021 @08:03PM (#26406561)
I really like what you have done.

One easy improvement "Showing results 16 — 30 out of 26 results" :-) showing below search results...

> Hope to include more curated signals, like "won a paper > prize", "journal in DOAJ and other reviewed indices", etc. This would be a great addition.

Re:Internet Archive Scholar: Search Millions of Research Papers by bnewbold on Wednesday March 10, 2021 @02:40AM (#26408835)
Fixed, thanks!
Re:Internet Archive Scholar: Search Millions of Research Papers by tomthe on Wednesday March 10, 2021 @06:26AM (#26410148)
Thank you for this great datasource!

Do you think this is suitable for bibliometric research? (We don't need citation-graphs). We use Scopus and Web of Science, but I really don't like that we are not able to publish helpful datasets that we extract from these databases.

Re:Internet Archive Scholar: Search Millions of Research Papers by bnewbold on Wednesday March 10, 2021 @04:59PM (#26416878)
I think it is in a good place for simple bibliometric queries. The fatcat elasticsearch API is open at https://api.fatcat.wiki/fatcat_release/ (behind a proxy to filter "unsafe" requests). That works pretty well for jupyter notebook style experimentation if you are willing to learn the elasticsearch query DSL for aggregations and things.

I don't think the catalog has high enough metadata quality today for use in published research. There are some glaring errors and omissions when you actually starting digging in. On the other hand, almost all bibliographic catalogs seem to have such problems. Fatcat, by being open and having an API, does have the potential to aggregate corrections, fixes, and contributions directly from researchers over time.

A particular missing piece today is that there is no categorization or "discipline" metadata of almost any type. This sort of metadata is more subjective, and the catalog currently carefully only includes factual information. We will likely start collecting metadata at the journal ("container") level and can trickle that down to papers. Aggregating, editing, and curating that metadata in Wikidata first, then importing to Fatcat, might be the best and most sustainable path forward.

Re:Internet Archive Scholar: Search Millions of Research Papers by fenrissan on Wednesday March 10, 2021 @11:19AM (#26412828)
Today I discovered "Open Access Diamond journals" in a report (https://zenodo.org/record/4558704/files/OADJS-Findings.pdf). These are low-scale peer-reviewed free-to-read free-to-publish non-commercial journals, typically supported by Universities or Goverment agencies. They serve diverse communities and are not predatory journals (they are free after all).

The bad news is only half of them use DOI or embed licenses in the metadata. Are they indexed or archived somewhere?

To my surprise, there are more than 350,000 papers published in OA Diamond journals every year, and most journals publish fewer than 25 articles a year.