SentenceTransformers: Python framework for sentence, text and image embeddings

Re:SentenceTransformers: Python framework for sentence, text... by anon373839 on Tuesday April 09, 2024 @06:40PM (#39985102)

This has got me curious. I don't really understand how binary labels could give us embeddings that are well-ordered by distance. For a training pair where (Q, D) are highly similar and a pair where (Q, D) are just barely related, the model is being trained that they're the same distance apart. Is there something I'm not seeing here?

[ Parent | on HN ]

Re:SentenceTransformers: Python framework for sentence, text... by jmalicki on Wednesday April 10, 2024 @03:03PM (#39994507)

See contrastive losses and siamese networks, like here (where they use L2 distance):

https://lilianweng.github.io/posts/2021-05-31-contrastive/

If documents are similar, you want the two embeddings to be close to each other, if they are dissimilar, you want them to be far apart.

Binary relevance judgements of course don't necessarily produce an ordering, but usually over a large enough set of training examples some will be better matches than others.

"Learning to Rank" gets you into all kinds of labels and losses if you want to go down that rabbit hoel.

[ Parent | on HN ]