Re:Semantic search engine for ArXiv, biorxiv and medrxiv
by heisenburgzero on Tuesday May 20, 2025 @11:19PM
(#44047982)
So did you just combine Title+Abstracts+Authors into a single chunk and embed them or embedded them individually?
Re:Semantic search engine for ArXiv, biorxiv and medrxiv
by synctext on Wednesday May 21, 2025 @01:17AM
(#44048496)
Impressive!
Will you parse the papers in the future? Without citations this is not that usable for professors or scientists in general. The relevance ranking largely depends on showing these older, prominent papers.
(from our lab experience building decentralised search using transformers)
Re:Semantic search engine for ArXiv, biorxiv and medrxiv
by 0101111101 on Wednesday May 21, 2025 @10:25AM
(#44051773)
One chunk embedded together
Re:Semantic search engine for ArXiv, biorxiv and medrxiv
by elliotec on Wednesday May 21, 2025 @02:52AM
(#44048941)
Thank you!!
Re:Semantic search engine for ArXiv, biorxiv and medrxiv
by cluckindan on Wednesday May 21, 2025 @04:46AM
(#44049524)
That method can break when author names and subject matter collide.
Re:Semantic search engine for ArXiv, biorxiv and medrxiv
by 0101111101 on Wednesday May 21, 2025 @10:27AM
(#44051794)
True, but similarly if your embeddings are any good they'll capture interesting associations between authors, topics and your search query. If you find any interesting author overlap results I'd be very interested!
Re:Semantic search engine for ArXiv, biorxiv and medrxiv
by cluckindan on Wednesday May 21, 2025 @04:27PM
(#44055958)
Not exactly what I was looking for, but interesting nonetheless: https://arxivxplorer.com/?q=exotic+penis
I'm also maintaining a dataset of all the embeddings on kaggle if you want to use them yourself: https://www.kaggle.com/datasets/tomtum/openai-arxiv-embeddin...