Semantic search engine for ArXiv, biorxiv and medrxiv

Re:Semantic search engine for ArXiv, biorxiv and medrxiv by nblgbg on Tuesday May 20, 2025 @10:15PM (#44047705)

Just curious, are there any techniques other than using embeddings, computing cosine similarity, and sorting the results based on that? RRF could be used but again its very simple as well.

[ Parent | on HN ]

Re:Semantic search engine for ArXiv, biorxiv and medrxiv by forrestp on Wednesday May 21, 2025 @04:02AM (#44049302)

My understanding is that your levers are roughly better / more diverse embeddings or computing more embeddings (embed chunks / groups / etc) + aggregating more cosine similarities / scores. More flops = better search w/ steep diminishing returns

Colbert being a good google-able application of utilizing more embeddings.

Search ends up often being a funnel of techniques. Cheap and high recall for phase 1 and ratchet up the flops and precision in subsequent passes on the previous result set.

[ Parent | on HN ]

Re:Semantic search engine for ArXiv, biorxiv and medrxiv by 0101111101 on Wednesday May 21, 2025 @10:25AM (#44051769)

Exactly! A near property of the matryoshka embeddings is that you can compute a low dimension embedding similarity really fast and then refine afterwards.

[ Parent | on HN ]