Understanding BERT and Search Relevance (2019)

Re:Understanding BERT and Search Relevance (2019) by binarymax on Wednesday April 22, 2020 @11:08AM (#22946347)

Good question. Averaging can work for short documents, but this becomes less useful as the document text gets longer. When dealing with documents of multiple paragraphs, I've been gravitating more towards thinking "whats the most relevant part of the document?". And for this SBERT works very well - as you can pluck the document's most similar sentence(s) for the given query, and use that as a signal for the most relevant piece of information.

Of course there are many caveats here, and this is by no means a solved problem, but works well for queries that are more than a couple terms.

[ Parent | on HN ]

Re:Understanding BERT and Search Relevance (2019) by petulla on Wednesday April 22, 2020 @11:58AM (#22946955)

I'm not sure I understand in practice how that would work. You have sentence embeddings and a query embedding. You're comparing groups of sentences from a document to the query. But how do you decide how to group the sentences? Are you just up-front grouping similar sentences somehow?

[ Parent | on HN ]

Re:Understanding BERT and Search Relevance (2019) by binarymax on Wednesday April 22, 2020 @12:18PM (#22947204)

Building upon @m_ke's note, when you have the sentence embeddings indexed, you use query similarity with those sentences as an additional signal that can be used further. For example, you can weight sentences that appear earlier in the document higher, you can take the top N sentences and use those, you can apply clustering to find hotspots, etc.

I'd treat it as a base signal like BM25 for a field - which is frequently tuned and used as one of many features for relevance.

[ Parent | on HN ]

Re:Understanding BERT and Search Relevance (2019) by petulla on Wednesday April 22, 2020 @01:11PM (#22947826)

Sort of makes sense. You'd need a ground-truth search->document dataset, I think, to test on, though. Seems a bit hand-tuned.

[ Parent | on HN ]

Re:Understanding BERT and Search Relevance (2019) by binarymax on Wednesday April 22, 2020 @01:33PM (#22948061)

Absolutely! We call it a "judgement list" in search relevance land, and one is required and used for training and testing of a relevance config/model...though the training is often manual, until your configuration and infrastructure are mature enough for learning-to-rank.

[ Parent | on HN ]

Re:Understanding BERT and Search Relevance (2019) by binarymax on Wednesday April 22, 2020 @02:08PM (#22948445)

@petulla we've reached the thread depth limit so I can't reply to your last question, but typically MSMarco is used for SOTA testing of deep learning for search: https://microsoft.github.io/msmarco/

[ Parent | on HN ]

Re:Understanding BERT and Search Relevance (2019) by petulla on Wednesday April 22, 2020 @01:57PM (#22948318)

Last q: which datasets of this kind are used for SOTA testing or document clustering?

[ Parent | on HN ]

Re:Understanding BERT and Search Relevance (2019) by m_ke on Wednesday April 22, 2020 @12:02PM (#22947010)

You index an embedding for each sentence and match against sentence embeddings, then rank documents based on highest scoring sentence matches.

[ Parent | on HN ]

Re:Understanding BERT and Search Relevance (2019) by petulla on Wednesday April 22, 2020 @12:12PM (#22947123)

Seems problematic if the term entities/topics are spread across sentences.

[ Parent | on HN ]

Re:Understanding BERT and Search Relevance (2019) by m_ke on Wednesday April 22, 2020 @12:20PM (#22947233)

It really depends on your application but you can come up with a lot of different aggregation schemes to rerank top results based on the whole document.

[ Parent | on HN ]