Re:How to Write a Spelling Corrector
by raffi on Friday June 19, 2009 @06:05PM
(#665875)
That is what I ended up doing. Except this is raw data. Usually NLP researchers work from corpora that are already segmented into sentences and tagged with part-of-speech tags. I had to go through the hassle of segmenting and marking up the Wikipedia and Project Gutenberg.
[ Parent
| on HN ]
Would you consider making it available to others?
Perhaps the UCI Machine Learning Repository would accept it. If not, I'm sure I could find a way to have it hosted at my university.
archive.org will host datasets.
To preprocess wikipedia, I have used the following software: http://sourceforge.net/apps/mediawiki/wikiprep/index.php?tit...
To remove boilerplate from gutenberg requires painfully constructed heuristics. It would be great to have software released to do that.