Re:Neural Search for medium sized corpora
by agencies on Monday January 10, 2022 @01:00PM
(#29878742)
How big is a medium sized corpus? 10m documents?
[ Parent
| on HN ]
Re:Neural Search for medium sized corpora
by raphaelty on Monday January 10, 2022 @01:40PM
(#29879422)
I think 10 million documents is a large corpus. A retriever like Sklearn TfIdf will have a hard time handling it in a reasonable time. The main goal of Cherche is to prototype a neural search engine quickly and with a large choice of retrievers and rankers for corpus sizes < 1 million documents which is a common use case in the industry.
Search implements a wrapper of the Python ElasticSearch client that is scalable and dedicated to corpora composed of tens of millions of documents.