SentenceTransformers: Python framework for sentence, text and image embeddings

Re:SentenceTransformers: Python framework for sentence, text... by pietz on Sunday April 07, 2024 @01:11PM (#39962144)

I haven't noticed this behavior in such extreme ways as you. I've worked with many different embeddings using this library and the OpenAI API for data normalization, extraction and structuring tasks.

On one hand it's super impressive what you can get out of these embeddings. On the other I'm with you that there seems to be a missing piece to make them work great.

An additional MLP or Siamese Network works really well but it seems like there should be something easier and unsupervised given your set of vectorized samples.

Thinking out loud, could PCA help with this by centering the data and removing dimensions that mostly model noise?

[ Parent | on HN ]

Re:SentenceTransformers: Python framework for sentence, text... by anon373839 on Sunday April 07, 2024 @06:38PM (#39964578)

> I haven't noticed this behavior in such extreme ways as you.

I should have clarified that it really depends on the model. I have had this problem to a greater extent with the GTE and BGE embeddings. But I use them anyway, because they’re so strong overall.

PCA is an interesting idea, and worth looking at.

[ Parent | on HN ]

Re:SentenceTransformers: Python framework for sentence, text... by whakim on Monday April 08, 2024 @05:59PM (#39974176)

I've tried PCA (and perhaps more usefully, UMAP) and while can be great for stuff like unsupervised learning, I think the trick is to choose a classifier (as per my comment above) that can do a good job with high-dimensional data.

[ Parent | on HN ]