Is Cosine-Similarity of Embeddings Really About Similarity?

Re:Is Cosine-Similarity of Embeddings Really About Similarity? by superkuh on Tuesday March 12, 2024 @12:31AM (#39676212)

That might be true for one-hot vectors but it's not true for learned embedding through the lens of attention. That said, I only made to page 3/9 of the paper before the mark-up for the math went over my head.

[ Parent | on HN ]

Re:Is Cosine-Similarity of Embeddings Really About Similarity? by latency-guy2 on Tuesday March 12, 2024 @02:14AM (#39676666)

If we're talking about adding dimensionality and relying on the kernel, sure, but I only get so much incorrect by going from 1 to M dims.

I can't of course be certain in all cases, but dimensions are typically (past experience, and using knowledge from word2vec experiments from years ago) derivative of higher dimensions. The kernel still operates on the same concept by applying a norm along with whatever weightings to each dim.

Semantic understanding is still not there in my opinion, we might feign it by increasing specificity, but only so much. Largest contributor will likely still the determining factor rather than the series of smaller, more specific dimensions.

I tested this using similar sentences as my original comment and failing in more scenarios than passing. I of course am biased since it may be given I did not select the right dimensions or measures.

[ Parent | on HN ]

Re:Is Cosine-Similarity of Embeddings Really About Similarity? by seanbethard on Tuesday March 12, 2024 @01:55AM (#39676585)

It is true. And if you want to say anything about meaning this isn't even the right math.

[ Parent | on HN ]