Re:Google 5gram corpus has unreasonable 5grams
by Natsu on Wednesday June 16, 2010 @02:30AM
(#1434892)
Dwarf Fortress could account for some part of that, too, even though trolls don't show up until later in the game. Actually, there are probably quite a few games and such where that sort of thing could happen. But I'm guessing that most of the weird data came from spammers, even if I think that some of those are real.
It's too bad they can't get n-grams only from published books. That would get rid of most of the spam, provided they filter out that book spammer who reprocesses Wikipedia into gibberish-filled books.
It's too bad they can't get n-grams only from published books. That would get rid of most of the spam, provided they filter out that book spammer who reprocesses Wikipedia into gibberish-filled books.