Re:A Guide to Natural Language Processing
by pencilcode on Wednesday November 15, 2017 @07:21AM
(#15703236)
Regarding finding similar documents what is the state of the art nowadays, LDA, word2vec, something else? What do you normally use?
[ Parent
| on HN ]
Re:A Guide to Natural Language Processing
by zintinio5 on Wednesday November 15, 2017 @03:29PM
(#15707447)
Like everything else, depends on your use-case. I have personally used TF-IDF vectors and token sets with Cosine and Jaccard distances in practice.
Some examples of use-cases: are you searching for "semantically similar", or "near duplicate"? You can compare documents under different metrics and different _representations_. Some representations are: LSA, PLSA, LDA, TF-IDF, and Set representations, along with metrics such as Jaccard Distance, Cosine Distance, Euclidean distance, etc.
Doc2vec is the Word2vec analog for documents.
Word Mover Distance on Word2Vec vectors.
There is an implementation in Textacy.
Re:A Guide to Natural Language Processing
by nicklovescode on Wednesday November 15, 2017 @01:45PM
(#15706537)
Have you heard of word mover’s distance? It works really well!