News for hackers. Stuff that matters.
Re:Hacker News' Reading Level by nl on Sunday December 12, 2010 @10:24PM (#1999132)
I'd be lying if I said I don't doubt you are not incorrect ;)

What you are proposing is a statistically generated version of the Gunning Fog Index (http://en.wikipedia.org/wiki/Gunning_fog_index) or the Flesch–Kincaid test (http://en.wikipedia.org/wiki/Flesch%E2%80%93Kincaid_readabil...).

If I were Google, I'd try that, but I'd also try something like working out percentage deviation from a Markov chain generated from their crawl. A method like that would show that my first sentence is pretty unreadable, while an algorithm based on word complexity would see it as pretty simple.

Re:Hacker News' Reading Level by biotech on Monday December 13, 2010 @02:09AM (#1999680)
> I'd be lying if I said I don't doubt you are not incorrect

Holy Shnikes, that's a tough one to parse! I wasn't sure what you were saying here, so I'm gonna break it down, working from the end of the sentence:

1. I'd be lying if I said I don't doubt you are not incorrect

2. I'd be lying if I said I don't doubt you are CORRECT

3. I'd be lying if I said I don't think you are INCORRECT

4. I'd be lying if I said I think you are CORRECT

5. I think you are INCORRECT

The idea is that each of the previous statements are saying basically the same thing; I'm just cancelling negatives each time. Anyway, am I correct to assume that you think that the GP is incorrect?

Re:Hacker News' Reading Level by billswift on Monday December 13, 2010 @02:19AM (#1999705)
Your transition 2 -> 3 is not justifiable, "doubt" is not the opposite of "think". A better parsing leaves "doubt" alone and would end with: "I doubt you are correct."
Re:Hacker News' Reading Level by gimpf on Monday December 13, 2010 @05:28AM (#2000031)
My dear, you just lost the "don't".
Re:Hacker News' Reading Level by oh_ryan on Monday December 13, 2010 @06:04AM (#2000074)
I'd be lying if I said I don't doubt you are not incorrect

I'd be lying if I said I don't doubt you are correct

I'd be lying if I said I think you are correct

Re:Hacker News' Reading Level by nl on Monday December 13, 2010 @05:34PM (#2002237)
I had to triple check to make sure I got it right, and I wrote it.

Yes, I think the GP is incorrect.

Re:Hacker News' Reading Level by derefr on Sunday December 12, 2010 @11:01PM (#1999251)
> If I were Google, I'd try that, but I'd also try something like working out percentage deviation from a Markov chain generated from their crawl.

Indeed, that was my second thought, but I wonder if the gains are really all that large over a raw statistical analysis of the word bag, and whether they're worth the extra analysis space/time. It really depends on what Google is planning on doing with this metadata, internally; if an order-of-ten precision is fine (to pick out decisive categorizations), the raw analysis may be all that's needed.

Re:Hacker News' Reading Level by nl on Sunday December 12, 2010 @11:12PM (#1999282)
Well we could always try it out. Here's 24GB (compressed) of ngram data from Google: http://googleresearch.blogspot.com/2006/08/all-our-n-gram-ar...