News for hackers. Stuff that matters.

A comment on NLP Challenge: Find semantically related terms over a large vocabulary (1M)?. Parent comment

Re:NLP Challenge: Find semantically related terms over a lar... by retube on Saturday November 06, 2010 @04:56AM (#1876497)
It's possible he's randomly generated the docs or something. Which, I agree, is a bit lame. It's nice to get optical confirmation that what you're doing is correct, and to get some real insight into language.

Much better would to have been take real sentences from wikipedia.

[ Parent | on HN ]
Re:NLP Challenge: Find semantically related terms over a lar... by bravura on Saturday November 06, 2010 @12:55PM (#1877249)
No. Grandparent has picked out a few pathological examples out of many good ones.

Here is a uniform, random sample of documents:

  faststretch details
  honourable judge
  oriental museum http
  london wc 1n 3 bg pdsg seminar
  1857-8 indian mutiny
  vera bell
  katherine r.b
  rare books collection
  novy arbat street
  universitys case
The data set is unique terms that occur in a crawl of .uk. Here is how I constructed the data set:

I took the UKWAC web-as-corpus crawl (2 bil­lion words, crawled in 2008), ran it through the splitta sen­tence split­ter, removed all funny char­ac­ters, ran the Penn tree­bank word tok­enizer, and per­form term extrac­tion with topia.termextract, dis­card­ing terms that will sin­gle words. I then low­er­cased the terms, sorted them, and uniqued them, to give the dataset.

[ Parent | on HN ]