News for hackers. Stuff that matters.
Re:Hacker News' Reading Level by derefr on Sunday December 12, 2010 @11:01PM (#1999251)
> If I were Google, I'd try that, but I'd also try something like working out percentage deviation from a Markov chain generated from their crawl.

Indeed, that was my second thought, but I wonder if the gains are really all that large over a raw statistical analysis of the word bag, and whether they're worth the extra analysis space/time. It really depends on what Google is planning on doing with this metadata, internally; if an order-of-ten precision is fine (to pick out decisive categorizations), the raw analysis may be all that's needed.

Re:Hacker News' Reading Level by nl on Sunday December 12, 2010 @11:12PM (#1999282)
Well we could always try it out. Here's 24GB (compressed) of ngram data from Google: http://googleresearch.blogspot.com/2006/08/all-our-n-gram-ar...