News for hackers. Stuff that matters.
Re:How to Write a Spelling Corrector by lsb on Friday June 19, 2009 @05:39PM (#665842)
Start with Wikipedia? They've got a few billion words there, and it's a 5GB download.
Re:How to Write a Spelling Corrector by aquarin on Friday June 19, 2009 @06:16PM (#665884)
Yes, I done exactly this. it is actually 20GB uncompressed and takes 5-6h to parse. http://appraisalmind.com/articles/wikipedia-word-frequency-l...
Re:How to Write a Spelling Corrector by raffi on Friday June 19, 2009 @06:05PM (#665875)
That is what I ended up doing. Except this is raw data. Usually NLP researchers work from corpora that are already segmented into sentences and tagged with part-of-speech tags. I had to go through the hassle of segmenting and marking up the Wikipedia and Project Gutenberg.
Re:How to Write a Spelling Corrector by mattrepl on Friday June 19, 2009 @06:18PM (#665885)
Would you consider making it available to others?

Perhaps the UCI Machine Learning Repository would accept it. If not, I'm sure I could find a way to have it hosted at my university.

Re:How to Write a Spelling Corrector by bravura on Friday June 19, 2009 @07:47PM (#665958)
archive.org will host datasets.

To preprocess wikipedia, I have used the following software: http://sourceforge.net/apps/mediawiki/wikiprep/index.php?tit...

To remove boilerplate from gutenberg requires painfully constructed heuristics. It would be great to have software released to do that.

Re:How to Write a Spelling Corrector by Create on Saturday June 20, 2009 @04:39AM (#666272)
This is why GOOG silently backed Wikipedia: to gain quality content regarding syntax and semantics. Other than give people a "good reason to surf" and hit "ads" [trackers] down the road.