News for hackers. Stuff that matters.
Re:Google 5gram corpus has unreasonable 5grams by wrs on Wednesday June 16, 2010 @01:01AM (#1434761)
It's funny that the comments on this post have been spammed with the sort of duplicative trash he may be seeing in the corpus. I never imagined that spam comments could actually be relevant.
Re:Google 5gram corpus has unreasonable 5grams by ubernostrum on Wednesday June 16, 2010 @12:02AM (#1434658)
The trolls/dwarfs one is almost certainly due to the works of Terry Pratchett.
Re:Google 5gram corpus has unreasonable 5grams by ZeroGravitas on Wednesday June 16, 2010 @05:08AM (#1435083)
Most of the weird ones are explained in the comments, you just have to dig through some spam.

* The trolls/dwarf one was part of the blurb repeated in bookstores and reviews for a Terry Pratchett novel

* the poet wicked the woman was part of a list of plays with "Wicked" (about the witch from Oz) being in the middle

* The prince compiled the Mishna is apparently something from Jewish lore.

I'd guess there'd be similar explanations for the others just like "the matrix reloaded the matrix" which he calls out, just not as obvious.

One of the commenters points out that any string of 5 words is already an outlier so you're naturally going to get noise like this mixed in.

If I google the first one it seems to be some standard boilerplate text used in collecting data about chemical exposure. The second is a list of UK reality TV series "Popstars: The Rivals, Shattered, The Farm". So they are widely used on the web, just not in normal speech.

Re:Google 5gram corpus has unreasonable 5grams by Natsu on Wednesday June 16, 2010 @02:30AM (#1434892)
Dwarf Fortress could account for some part of that, too, even though trolls don't show up until later in the game. Actually, there are probably quite a few games and such where that sort of thing could happen. But I'm guessing that most of the weird data came from spammers, even if I think that some of those are real.

It's too bad they can't get n-grams only from published books. That would get rid of most of the spam, provided they filter out that book spammer who reprocesses Wikipedia into gibberish-filled books.

Re:Google 5gram corpus has unreasonable 5grams by Caligula on Tuesday June 15, 2010 @10:57PM (#1434552)
I am glad to find out there are some other issues with it. My main disappointment with it however remains its license. It would of been really cool if Google released it with a CC or MIT license but instead its restricted for academic usage only. Better to spend time with other corpus'.
Re:Google 5gram corpus has unreasonable 5grams by steveitis on Wednesday June 16, 2010 @01:35AM (#1434816)
What other n-gram corpus' are there? I've been looking, and all I can seem to find is some web spammer trying to sell me one.

I don't particularly care about the license, but I'd rather not have to build my own spider to crawl and generate one for me.

Re:Google 5gram corpus has unreasonable 5grams by GFischer on Wednesday June 16, 2010 @09:05AM (#1435414)
Re:Google 5gram corpus has unreasonable 5grams by donaldc on Wednesday June 16, 2010 @03:32AM (#1434963)
Better yet if they also released the corpus over bittorrent instead of requiring one spend $100+ for the cds. Let everyone play with it, not just full-time researchers.
Re:Google 5gram corpus has unreasonable 5grams by gojomo on Wednesday June 16, 2010 @01:20AM (#1434794)
Quite possibly the crawler collecting the corpus hit a crawler trap (intentional or unintentional) -- or perhaps web-based game output (which when visited by a crawler became a de facto trap) -- which multiplied the implausible phrases.