News for hackers. Stuff that matters.
Re:Neural Search for medium sized corpora by gk1 on Monday January 10, 2022 @08:23AM (#29874377)
Great to see another neural search (better known as vector search) option, especially with an end-to-end pipeline. There were 3 good discussions on vector search about a month ago:

https://news.ycombinator.com/item?id=29551947 - About semantic search (ie, vector search applied to text)

https://news.ycombinator.com/item?id=29554986 - About vector search at Google

https://news.ycombinator.com/item?id=29555780 - Open-source vector search index from Facebook

Re:Neural Search for medium sized corpora by raphaelty on Monday January 10, 2022 @09:39AM (#29875450)
Thank you for these great resources.
Re:Neural Search for medium sized corpora by akie on Monday January 10, 2022 @07:52AM (#29874020)
If you're confused (like me) and don't know what "neural search" is, here's an explainer https://medium.com/jina-ai/what-is-jina-and-neural-search-7a... (scroll down a bit)
Re:Neural Search for medium sized corpora by throwaway81523 on Tuesday January 11, 2022 @06:17AM (#29890044)
That article is very basic and fluffy, but the more technical article that it is based on and links to is excellent, so thanks for (indirectly) pointing to it:

https://hanxiao.io/2018/01/10/Build-Cross-Lingual-End-to-End...

The other stuff on that guy's (Han Xiao's) blog also looks interesting. I think this neural search stuff explains why we're all getting such low-precision results on Google/DDG by the standards we were used to. But, the claimed advantages are impressive. This is a big change in how search engines work, and I was unfamiliar with it before.

I still have reservations about where enough training data is supposed to come from, without already having a big system and user base. But, maybe some more info about that is out there. So this is another thing to look into. Wow.

Re:Neural Search for medium sized corpora by gk1 on Monday January 10, 2022 @08:18AM (#29874303)
“Vector search” is more commonly used, so you might’ve heard it called that. In case not: https://www.pinecone.io/learn/what-is-similarity-search/
Re:Neural Search for medium sized corpora by agencies on Monday January 10, 2022 @01:00PM (#29878742)
How big is a medium sized corpus? 10m documents?
Re:Neural Search for medium sized corpora by raphaelty on Monday January 10, 2022 @01:40PM (#29879422)
I think 10 million documents is a large corpus. A retriever like Sklearn TfIdf will have a hard time handling it in a reasonable time. The main goal of Cherche is to prototype a neural search engine quickly and with a large choice of retrievers and rankers for corpus sizes < 1 million documents which is a common use case in the industry.

Search implements a wrapper of the Python ElasticSearch client that is scalable and dedicated to corpora composed of tens of millions of documents.