Indexing Billions of Text Vectors

Re:Indexing Billions of Text Vectors by visarga on Sunday December 08, 2019 @01:43AM (#21734663)

I tried weighted average, but in my case the weights were computed by doing dot product between all vectors in the phrase (m x m), then taking the average over rows for each word and normalising. Kind of like a poor man's Transformer. It will boost words that are supported by other similar words in the same phrase.

[ Parent | on HN ]

Re:Indexing Billions of Text Vectors by Der_Einzige on Sunday December 08, 2019 @03:53AM (#21734984)

Can you provide psudocode or code for this?

I'd post a snippet of my implementation but I'm too dumb to figure out the format.

But wow this is much better for results! Now to figure out how to make it fast!

[ Parent | on HN ]

Re:Indexing Billions of Text Vectors by visarga on Monday December 09, 2019 @01:03PM (#21744806)

Something like this, including thresholding for pairs of words that have low dot product:

    import numpy as np

    def inner_product_rank(vecs, threshold=0.5):
        sims_mat = np.dot(vecs, vecs.transpose())
        sims_mat = sims_mat - threshold
        sims_mat[sims_mat<0] = 0
        ranks = np.sum(sims_mat, axis=0)
        return ranks

Then you can take a weighted sum of the vecs or use the ranks to select the most related words. It is also possible to run spectral clustering on sims_mat to get the main topics of the text, it works quite well.

[ Parent | on HN ]