Re:How to Get into Natural Language Processing
by demonshalo on Saturday January 21, 2017 @07:39AM
(#13449810)
I started even simpler than that. I started by just eliminating stopwords and count the frequency in each word in the document itself. I did not use a set of documents as the goal was for the algorithm to be used on the spot for a single block of text.
A few months later and after many iterations + a whole lot of testing, the algorithm now can extract super relevant keywords 90%+ of the time!
I wish I knew about the WikiCorpusExtractor. Thanks for the link!
The only thing one really needs is to count the frequency of words in each document and do very simple math. Tf-Idf is still very relevant today and provides you with a very good idea on how statistics is used on text-mining.
[0] https://en.wikipedia.org/wiki/Tf%E2%80%93idf
[1] https://github.com/joaoventura/WikiCorpusExtractor