News for hackers. Stuff that matters.

A comment on Dense Vector and Sparse Vector and Fulltext and Tensor Reranker = Best for RAG?. Parent comment

Re:Dense Vector and Sparse Vector and Fulltext and Tensor Re... by yingfeng on Tuesday July 16, 2024 @10:17PM (#40982016)
Hi, I'm one of the creators of infinity, and the article has mentioned about the sparse vector vs bm25. While the sparse vector performs well under some evaluations, it is obtained by training a model, which means that it can't fully represent all of the user's keywords/tokens, and those that don't appear in the training set, are truncated. So this is a very big impact for many enterprise vertical scenarios. And bm25 doesn't have such a limitation
[ Parent | on HN ]
Re:Dense Vector and Sparse Vector and Fulltext and Tensor Re... by philippemnoel on Wednesday July 17, 2024 @08:11AM (#40984975)
BM25 is indeed way more important than these vector DBs will claim. At ParadeDB, we've observed significant use cases where customers need both
[ Parent | on HN ]