News for hackers. Stuff that matters.

A comment on 78% of Reddit Threads with 1,000+ Comments Mention Nazis.

Re:78% of Reddit Threads with 1,000+ Comments Mention Nazis by Someone on Wednesday May 04, 2016 @02:50PM (#11630686)
So, is that a lot? Let’s assume a very simple model, with Reddit writers picking words from a fixed distribution, 10 words per reply, and all ‘over 1,000 comments’ threads having exactly 1000 comments.

Then, if the probability of picking ‘fried rice’ is p, the probability of ‘fried rice’ not appearing in that 1,000 comment, 10,000 word thread is

  (1-p)^10000
That apparently, is around 0.22:

  (1-p)^10000 = 0.22
Solving for p gives us

  10000 log(1-p) = log(.22)

  log(1-p) = log(.22)/10000

  1-p = exp(log(.22)/10000)

  p = 1 - exp(log(.22)/10000)
That’s about 0.000066, or 1 in 15,000.

Let’s compare that with https://books.google.com/ngrams/graph?content=nazi&year_star....

That gives us numbers close to 0.000066, but only for the mid 1940’s. Current level is around 0.000008 or 1 in 125,000. Let’s compute how many words such a long Reddit thread needs to get at that level:

  (1-0.000008)^w = 0.22

  w log(1-0.000008) = log(0.22)

  w = log(0.22)/log(1-0.000008)
That gives me around 190.000 words. That does not seem impossible for me. If long threads on average have 2,500 comments, it is only 36 words per comment. One would have to look at the distribution of thread length and #words/comment to say more about this.

Now, let’s hope there isn’t an embarrassing error in the above computations :-)

[ Parent | on HN ]
Re:78% of Reddit Threads with 1,000+ Comments Mention Nazis by rossdavidh on Wednesday May 04, 2016 @04:36PM (#11631538)
I think it would be a better comparison to simply pick 100 other words, perhaps proper nouns like "Communist", "Stalin", "Fascist", "Mussolini", "Khmer Rouge", "Pol Pot", and etc. Do the same analysis with these, and it will tell you if Godwin's Law is telling us something unique about this particular bugbear. (I suspect it is, but haven't run the comparison myself)
[ Parent | on HN ]
Re:78% of Reddit Threads with 1,000+ Comments Mention Nazis by coldtea on Wednesday May 04, 2016 @04:57PM (#11631729)
Well, all those are simple regimes or dictators etc, whereas Nazi has been relegated to stand in for the ultimate evil, or as a synonym for evil itself.
[ Parent | on HN ]
Re:78% of Reddit Threads with 1,000+ Comments Mention Nazis by Aelinsaar on Wednesday May 04, 2016 @06:23PM (#11632418)
Especially organized, bureaucratic, systemic, and industrialized evil.
[ Parent | on HN ]
Re:78% of Reddit Threads with 1,000+ Comments Mention Nazis by paulddraper on Wednesday May 04, 2016 @08:58PM (#11633207)
Lawful evil
[ Parent | on HN ]
Re:78% of Reddit Threads with 1,000+ Comments Mention Nazis by Aelinsaar on Thursday May 05, 2016 @03:06PM (#11638880)
Led by a core of Chaotic Evil people. "Invade Russia!" "Exterminate the Jews!" all starts to scream CE over LE at the leadership level.
[ Parent | on HN ]
Re:78% of Reddit Threads with 1,000+ Comments Mention Nazis by paulddraper on Thursday May 05, 2016 @04:31PM (#11639445)
Eh, that's like almost all leaders though.

"Give me liberty or death!"

"Invade Mexico!"

"Assassinate Castro!"

"Go to the moon!"

When you're the one (at least in part) making the laws, it's hard to distinguish your decisions been between chaos and lawful.

[ Parent | on HN ]
Re:78% of Reddit Threads with 1,000+ Comments Mention Nazis by tatterdemalion on Thursday May 05, 2016 @01:55AM (#11634247)
That's the entire point of Godwin's Law.
[ Parent | on HN ]
Re:78% of Reddit Threads with 1,000+ Comments Mention Nazis by minimaxir on Wednesday May 04, 2016 @03:17PM (#11630925)
Bernoulli trials assume events are independent. That is likely not a good assumption in this case.
[ Parent | on HN ]
Re:78% of Reddit Threads with 1,000+ Comments Mention Nazis by Bartweiss on Wednesday May 04, 2016 @04:16PM (#11631395)
It seems like a terrible assumption for this case.

Godwin's Law is largely a specific case of conversational drift (unless you start out talking about Nazis). Everyone starts out on topic, then gets more hyperbolic and distracted until you end up at some other idea, in this case Nazis.

So not only are comments in threads not independent samples, we're specifically looking at a proposition about non-independence.

[ Parent | on HN ]
Re:78% of Reddit Threads with 1,000+ Comments Mention Nazis by wiml on Wednesday May 04, 2016 @11:04PM (#11633744)
He's formulating a null hypothesis - a (perhaps rejectable) alternative to the unspoken assumption in most of this thread, which is that Reddit, or perhaps comment threads in general, are disproportionately likely to include a mention of Nazis as they grow.
[ Parent | on HN ]
Re:78% of Reddit Threads with 1,000+ Comments Mention Nazis by ececconi on Wednesday May 04, 2016 @04:33PM (#11631517)
People don't pick words from a fixed distribution in the manner which you wrote. I scanned over 500,000 words I wrote and I never mentioned the word mentioned on the headline of this article. I'm not sure if any kind of meaningful interpretation can be made from the headline, but I think it's an interesting observation nonetheless.
[ Parent | on HN ]
Re:78% of Reddit Threads with 1,000+ Comments Mention Nazis by Someone on Wednesday May 04, 2016 @06:01PM (#11632280)
People do not do that, but at least it is a model that one can improve. For example, as other commenters indicate, one can argument that occurrences within threads will be correlated (that will happen in discussions about World War Two, for example) but others indicated that "Godwin's law" occurrences may follow the model better.

In contrast, the OP just presented a bare number, implying that it was high, but without any evidence for that claim.

And yes, there's lots of room for improvement. The model is very simple, the guesses at the Reddit thread sizes are 100% guesses (how long is the average comment? What's the distribution of thread lengths? Etc), I did some serious rounding here and there, trusted Google to do the math right (it is not that hard, but IEEE may be insufficient to get results that are somewhat correct, and I don't know what Google does), etc.

As to your "I never wrote nasi goreng in 500,000 words" argument: let's take my simple/flawed model. It predicts

  (1-1/125000)^500000 ~= 0.018
as the probability that 500,000 words do not contain "taboo". So, that simple/rough/flawed model predicts that, out of 1000 people writing 500,000 words, 18 wouldn't mention "Voldemort". You could easily be one of them. It's about as likely as somebody flipping 6 heads in a row. So, I don't think that single data point invalidates the model (no, I haven't done the proper statistics)
[ Parent | on HN ]
Re:78% of Reddit Threads with 1,000+ Comments Mention Nazis by hrayr on Wednesday May 04, 2016 @04:44PM (#11631601)
> I scanned over 500,000 words I wrote and I never mentioned the word mentioned on the headline of this article.

Funny how you're trying to avoid breaking your streak of not mentioning nazis.

[ Parent | on HN ]
Re:78% of Reddit Threads with 1,000+ Comments Mention Nazis by ececconi on Wednesday May 04, 2016 @04:45PM (#11631616)
Yep, a little joke.
[ Parent | on HN ]
Re:78% of Reddit Threads with 1,000+ Comments Mention Nazis by fullshark on Wednesday May 04, 2016 @05:59PM (#11632265)
He meant "reddit"
[ Parent | on HN ]
Re:78% of Reddit Threads with 1,000+ Comments Mention Nazis by webkike on Wednesday May 04, 2016 @03:13PM (#11630884)
Where did you get that the probability of a thousand word comment not containing the fried rice is 0.22?
[ Parent | on HN ]
Re:78% of Reddit Threads with 1,000+ Comments Mention Nazis by 0xffff2 on Wednesday May 04, 2016 @03:19PM (#11630941)
"Fried rice" is a standin for nazis/Hitler, so 1-0.78=0.22.
[ Parent | on HN ]
Re:78% of Reddit Threads with 1,000+ Comments Mention Nazis by WalterBright on Thursday May 05, 2016 @02:05AM (#11634271)
> 190.000 words

The average TV vocabulary is 2,000 words. High school graduates average a 10,000 word vocabulary, college graduates 30,000, and there are over a million words in the English language. Hence having 190,000 distinct words in a reddit thread seems implausible.

[ Parent | on HN ]
Re:78% of Reddit Threads with 1,000+ Comments Mention Nazis by valleyer on Thursday May 05, 2016 @02:50AM (#11634426)
They needn't be distinct. Writing is random selection with replacement.
[ Parent | on HN ]