Re:Google 5gram corpus has unreasonable 5grams
by ZeroGravitas on Wednesday June 16, 2010 @05:08AM
(#1435083)
Most of the weird ones are explained in the comments, you just have to dig through some spam.
* The trolls/dwarf one was part of the blurb repeated in bookstores and reviews for a Terry Pratchett novel
* the poet wicked the woman was part of a list of plays with "Wicked" (about the witch from Oz) being in the middle
* The prince compiled the Mishna is apparently something from Jewish lore.
I'd guess there'd be similar explanations for the others just like "the matrix reloaded the matrix" which he calls out, just not as obvious.
One of the commenters points out that any string of 5 words is already an outlier so you're naturally going to get noise like this mixed in.
If I google the first one it seems to be some standard boilerplate text used in collecting data about chemical exposure. The second is a list of UK reality TV series "Popstars: The Rivals, Shattered, The Farm". So they are widely used on the web, just not in normal speech.
* The trolls/dwarf one was part of the blurb repeated in bookstores and reviews for a Terry Pratchett novel
* the poet wicked the woman was part of a list of plays with "Wicked" (about the witch from Oz) being in the middle
* The prince compiled the Mishna is apparently something from Jewish lore.
I'd guess there'd be similar explanations for the others just like "the matrix reloaded the matrix" which he calls out, just not as obvious.
One of the commenters points out that any string of 5 words is already an outlier so you're naturally going to get noise like this mixed in.
If I google the first one it seems to be some standard boilerplate text used in collecting data about chemical exposure. The second is a list of UK reality TV series "Popstars: The Rivals, Shattered, The Farm". So they are widely used on the web, just not in normal speech.