News for hackers. Stuff that matters.
Re:NLP Challenge: Find semantically related terms over a lar... by bravura on Saturday November 06, 2010 @12:55PM (#1877249)
No. Grandparent has picked out a few pathological examples out of many good ones.

Here is a uniform, random sample of documents:

  faststretch details
  honourable judge
  oriental museum http
  london wc 1n 3 bg pdsg seminar
  1857-8 indian mutiny
  vera bell
  katherine r.b
  rare books collection
  novy arbat street
  universitys case
The data set is unique terms that occur in a crawl of .uk. Here is how I constructed the data set:

I took the UKWAC web-as-corpus crawl (2 bil­lion words, crawled in 2008), ran it through the splitta sen­tence split­ter, removed all funny char­ac­ters, ran the Penn tree­bank word tok­enizer, and per­form term extrac­tion with topia.termextract, dis­card­ing terms that will sin­gle words. I then low­er­cased the terms, sorted them, and uniqued them, to give the dataset.

There are no comments on this comment yet.