Re:“The Unreasonable Effectiveness of Deep Learning Represen...
by kujaomega on Friday July 06, 2018 @12:47PM
(#17472906)
I see a good explanation of the problem and a good evolution of the done steps. But I see a problem in the approach. When you are getting the most similar result, you are supposed to compute high cosine similarity between all the embeddings. If you have more than a billion of embeddings and the embeddings have 1k dimensions, it will take a lot of time. How would you solve this problem? Clustering the embeddings?
[ Parent
| on HN ]
Re:“The Unreasonable Effectiveness of Deep Learning Represen...
by bunderbunder on Friday July 06, 2018 @01:53PM
(#17473571)
There are off-the-shelf libraries like ANNOY and nmslib that index the vectors in a way that allows for fast (possibly approximate) nearest neighbors searches.