The gensim library now offers the 'Paragraph Vector' [1] approach to create vectors for longer ranges of text. It's available in a class named Doc2Vec – but I don't think that's what is being used here.
The Paragraph Vector approach can give interesting results for document-similarity, including similarity after certain 'algebraic'-like additions/subtractions of other topics/word-concepts. [2]
Re:Using Doc2Vec to Suggest SubReddits
by jmportilla on Thursday August 27, 2015 @11:51PM
(#10133657)
I used the gensim Doc2Vec implementation. You can check out some of the source code here: https://github.com/jmportilla/Reddit2Vec
Great work. I guess my question is - do you use 'averaging' of word vectors or the Chinese Restaurant process - to get to sub reddit vectors. You describe the Chinese Restaurant process as a "more sophisticated method" that you "can" use, but in my experiments with word2vec and reddit (https://github.com/utunga/gensimred) I quickly discovered that simple averaging just does not work. Averaging has this awful 'revert to mean' thing that turns all the paragraph vectors into a sort of bland gray goo where they are all the same.
If you did use Chinese Restaurant process (I love that phrase - brings back memories of an occasion at a Dim Sum restaurant where this almost literally happened) it'd be great to see any source code you may feel like releasing ;_) ... well, it can't hurt to ask..