News for hackers. Stuff that matters.
Re:A Guide to Natural Language Processing by pencilcode on Wednesday November 15, 2017 @07:21AM (#15703236)
Regarding finding similar documents what is the state of the art nowadays, LDA, word2vec, something else? What do you normally use?
Re:A Guide to Natural Language Processing by zintinio5 on Wednesday November 15, 2017 @03:29PM (#15707447)
Like everything else, depends on your use-case. I have personally used TF-IDF vectors and token sets with Cosine and Jaccard distances in practice.

Some examples of use-cases: are you searching for "semantically similar", or "near duplicate"? You can compare documents under different metrics and different _representations_. Some representations are: LSA, PLSA, LDA, TF-IDF, and Set representations, along with metrics such as Jaccard Distance, Cosine Distance, Euclidean distance, etc.

Doc2vec is the Word2vec analog for documents.

Re:A Guide to Natural Language Processing by nl on Wednesday November 15, 2017 @03:51PM (#15707616)
Word Mover Distance on Word2Vec vectors.

There is an implementation in Textacy.

Re:A Guide to Natural Language Processing by nicklovescode on Wednesday November 15, 2017 @01:45PM (#15706537)
Have you heard of word mover’s distance? It works really well!