News for hackers. Stuff that matters.
Re:Indexing Billions of Text Vectors by Der_Einzige on Sunday December 08, 2019 @03:53AM (#21734984)
Can you provide psudocode or code for this?

I'd post a snippet of my implementation but I'm too dumb to figure out the format.

But wow this is much better for results! Now to figure out how to make it fast!

Re:Indexing Billions of Text Vectors by visarga on Monday December 09, 2019 @01:03PM (#21744806)
Something like this, including thresholding for pairs of words that have low dot product:

    import numpy as np

    def inner_product_rank(vecs, threshold=0.5):
        sims_mat = np.dot(vecs, vecs.transpose())
        sims_mat = sims_mat - threshold
        sims_mat[sims_mat<0] = 0
        ranks = np.sum(sims_mat, axis=0)
        return ranks
Then you can take a weighted sum of the vecs or use the ranks to select the most related words. It is also possible to run spectral clustering on sims_mat to get the main topics of the text, it works quite well.