Re:How to Write a Spelling Corrector
by lsb on Friday June 19, 2009 @05:39PM
(#665842)
Start with Wikipedia? They've got a few billion words there, and it's a 5GB download.
[ Parent
| on HN ]
Yes, I done exactly this.
it is actually 20GB uncompressed and takes 5-6h to parse.
http://appraisalmind.com/articles/wikipedia-word-frequency-l...
That is what I ended up doing. Except this is raw data. Usually NLP researchers work from corpora that are already segmented into sentences and tagged with part-of-speech tags. I had to go through the hassle of segmenting and marking up the Wikipedia and Project Gutenberg.
Would you consider making it available to others?
Perhaps the UCI Machine Learning Repository would accept it. If not, I'm sure I could find a way to have it hosted at my university.
archive.org will host datasets.
To preprocess wikipedia, I have used the following software: http://sourceforge.net/apps/mediawiki/wikiprep/index.php?tit...
To remove boilerplate from gutenberg requires painfully constructed heuristics. It would be great to have software released to do that.