Re:English Letter Frequency Counts: Mayzner Revisited
by ComputerGuru on Saturday January 05, 2013 @01:49PM
(#5013544)
I would think tweets would be skewed, no? Slang, memes, shrtnd txt 2 avoid char lmts, http://urls/, amongst others?
Re:English Letter Frequency Counts: Mayzner Revisited
by talaketu on Sunday January 06, 2013 @02:46PM
(#5017447)
No corpus is immune from comparison, and each will have statistical parameters that reflect it's original selection criteria.
Perhaps Mayzner's corpus, apparently based on a sample from literature, exhibits a bias away from the abbreviated forms widely used in written communication today.
So, if you wanted to tune your text prediction software for your phone...
Re:English Letter Frequency Counts: Mayzner Revisited
by chime on Sunday January 06, 2013 @10:27PM
(#5019103)
Precisely. I was looking for predicting informal communication patterns, not formal book/newspaper style.
Re:English Letter Frequency Counts: Mayzner Revisited
by tcwc on Saturday January 05, 2013 @07:39AM
(#5012446)
Unfortunately Twitter no longer allow redistribution of tweet content (http://readwrite.com/2011/03/03/how_recent_changes_to_twitte...). Datasift and Gnip offer commercial access to the full firehose, but for a sample it wouldn't take long to build your own corpus from the official Twitter streaming or search API.
* http://ktype.net/wiki/research:articles:progress_20110209#tw...
* http://ktype.net/wiki/research:articles:progress_20110228?s#...
I would love to redo this analysis with newer Tweets but alas, don't know where to get a usable corpus. Any suggestions? My goal was to explore many of the concepts from norvig's http://norvig.com/spell-correct.html using the Tweet dataset to build a better one-finger-keyboard and word-prediction engine for iOS.