News for hackers. Stuff that matters.

A comment on Is Cosine-Similarity of Embeddings Really About Similarity?. Parent comment

Re:Is Cosine-Similarity of Embeddings Really About Similarity? by vl on Tuesday March 12, 2024 @11:56AM (#39681039)
While indeed modern embedding are more robust;

All embeddings are first layer of DNN. In case of word2vec this is shallow 2-layer network. Selection of embedding is multiplication of embedding matrix by one-hot vector, which is usually optimized as array lookup.

[ Parent | on HN ]
Re:Is Cosine-Similarity of Embeddings Really About Similarity? by minimaxir on Tuesday March 12, 2024 @01:09PM (#39682037)
That's not "all embeddings", that's just implementations like word2vec/fastText. And even though they are fast, they both don't get context as well and require significant preprocessing (e.g. stemming/stop word removal).

Implementations that use a LLM require a full forward pass, but the many optimizations in inference speed make it not noticable for small applications.

[ Parent | on HN ]