News for hackers. Stuff that matters.
Re:Indexing Billions of Text Vectors by visarga on Monday December 09, 2019 @01:03PM (#21744806)
Something like this, including thresholding for pairs of words that have low dot product:

    import numpy as np

    def inner_product_rank(vecs, threshold=0.5):
        sims_mat = np.dot(vecs, vecs.transpose())
        sims_mat = sims_mat - threshold
        sims_mat[sims_mat<0] = 0
        ranks = np.sum(sims_mat, axis=0)
        return ranks
Then you can take a weighted sum of the vecs or use the ranks to select the most related words. It is also possible to run spectral clustering on sims_mat to get the main topics of the text, it works quite well.

There are no comments on this comment yet.