Re:Generative Models
by 50CNT on Friday June 17, 2016 @03:40AM
(#11921172)
That reminds me of something I was doing recently, really rough corpus analysis, trying to see how much text coverage the words in the NGSL give you. Honestly thought it'd take me a couple of weeks to do.
Got into NLTK, used the built in sentence tokenizer, word tokenizer, then wordnet POS tagging to remove proper nouns, added some more cleanup code, and I had something passable within two days.
Now at this point I couldn't write a POS tagger to save my life, but it was cool seeing code you wrote over two evenings run over 30k books just like that (which still took a week, but ah well).
Got into NLTK, used the built in sentence tokenizer, word tokenizer, then wordnet POS tagging to remove proper nouns, added some more cleanup code, and I had something passable within two days.
Now at this point I couldn't write a POS tagger to save my life, but it was cool seeing code you wrote over two evenings run over 30k books just like that (which still took a week, but ah well).