Re:Language Models, Word2Vec, and Efficient Softmax Approxim...
by garysieling on Sunday October 29, 2017 @03:34PM
(#15580939)
If you wanted it to know that "machine learning" and "neural networks" were related, wouldn't you need to do some type of entity extraction first, since Word2vec is run on tokens?
[ Parent
| on HN ]
Re:Language Models, Word2Vec, and Efficient Softmax Approxim...
by visarga on Sunday October 29, 2017 @10:58PM
(#15582853)
You can use Gensim:
from gensim.models.phrases import Phrases
bigrams = Phrases(corpus)
or you could rank bigrams by count(w1+w2)^2/(count(w1)*count(w2))many variations on this formula work, but the idea is to compare the count of the bigram to the counts of the unigrams.
By the way, you do bigram identification before Word2Vec to have specialized vectors for bigrams as well.
Besides this method, there is one great way to identify ngrams: use Wikipedia titles. It's quite an extended list that covers most of the important named entities, locations and multi-word topic names, or go directly to http://wiki.dbpedia.org/ for a huge list with millions of ngrams. Cross reference it with your text corpus and you get a nice clean list.
Re:Language Models, Word2Vec, and Efficient Softmax Approxim...
by kbwt on Sunday October 29, 2017 @03:40PM
(#15580978)
The original word2vec source code comes with a probabilistic phrase detection tool. Keyword: word2phrase.
Re:Language Models, Word2Vec, and Efficient Softmax Approxim...
by garysieling on Sunday October 29, 2017 @07:01PM
(#15582028)
Good to know, thanks!