News for hackers. Stuff that matters.
Re:“The Unreasonable Effectiveness of Deep Learning Represen... by kujaomega on Friday July 06, 2018 @12:47PM (#17472906)
I see a good explanation of the problem and a good evolution of the done steps. But I see a problem in the approach. When you are getting the most similar result, you are supposed to compute high cosine similarity between all the embeddings. If you have more than a billion of embeddings and the embeddings have 1k dimensions, it will take a lot of time. How would you solve this problem? Clustering the embeddings?
Re:“The Unreasonable Effectiveness of Deep Learning Represen... by bunderbunder on Friday July 06, 2018 @01:53PM (#17473571)
There are off-the-shelf libraries like ANNOY and nmslib that index the vectors in a way that allows for fast (possibly approximate) nearest neighbors searches.
Re:“The Unreasonable Effectiveness of Deep Learning Represen... by bglazer on Friday July 06, 2018 @01:29PM (#17473342)
This is generally called the k-Nearest Neighbors problem. You should check out the various data structures for doing this, like the ball tree and kd tree.