BM25 – The Next Generation of Lucene Relevance

Posted by based2 on Sunday March 13, 2016 @12:07PM
from the opensourceconnections.com dept.

http://opensourceconnections.com/blog/2015/10/16/bm25-the-next-generation-of-lucene-relevation/

( 7 comments | 95 points | on HN )

Re:BM25 – The Next Generation of Lucene Relevance by asjo on Sunday March 13, 2016 @05:36PM (#11279457)

Xapian has been using BM25 forever[1] - what made Lucene switch now?

[1] https://xapian.org/docs/bm25.html

[ Parent | on HN ]

Re:BM25 – The Next Generation of Lucene Relevance by apatry on Sunday March 13, 2016 @07:14PM (#11279902)

BM25 has been available for Lucene since version 4 thanks to a refactoring that allowed anyone to plug in a custom relevance score if I remember correctly . The big news is that starting version 6, BM25 is the default.

[ Parent | on HN ]

Re:BM25 – The Next Generation of Lucene Relevance by asjo on Monday March 14, 2016 @12:37AM (#11281100)

Yes, so I am asking why is Lucene switching now?

[ Parent | on HN ]

Re:BM25 – The Next Generation of Lucene Relevance by nl on Monday March 14, 2016 @05:05AM (#11281741)

Here's the issue tracking[1].

Basically it looks like the typical conservative approach most large scale open source projects are forced to take because of cross dependencies.

It's a (potentially) breaking change, so they waited for a major release.

[1] https://issues.apache.org/jira/browse/LUCENE-6789

[ Parent | on HN ]

Re:BM25 – The Next Generation of Lucene Relevance by rpedela on Sunday March 13, 2016 @10:14PM (#11280638)

BM25 has been available for awhile so is there a reason it is becoming default soon besides it is better in certain scenarios? Have there been improvements to the implementation recently?

[ Parent | on HN ]

Re:BM25 – The Next Generation of Lucene Relevance by eva1984 on Monday March 14, 2016 @04:06AM (#11281583)

I went to the elastic conf last month, there is a great talk about BM25. It mentions that BM25 is more robust against common words than plain tf*idf.

https://www.elastic.co/elasticon/conf/2016/sf/improved-text-...

[ Parent | on HN ]

Re:BM25 – The Next Generation of Lucene Relevance by CephalopodMD on Sunday March 13, 2016 @11:54PM (#11280960)

Doesn't lucene let you choose your own relevance algorithms, smoothing, metrics, etc. anyway?

[ Parent | on HN ]