News for hackers. Stuff that matters.
Re:Geekiest Hacker News comments from the last month by riffer on Monday August 23, 2010 @03:51PM (#1627817)
TLDR version: We took a corpus of 25k comments from HN, analyzed them to infer semantic similarity. Next we came up with a seed of 18 words out of the 40k words in our corpus, scored the corpus of 40k words based on similarity with those 18 words. We then analyzed the 25k comments to score them based on the scores assigned to the 40k words. Major weaknesses: [1] words not in the corpus are dropped, so technical text with (relatively) obscure words may not score well, and [2] our seed of 18 words was highly subjective and put together in <5 mins.

Also, there are a couple of dynamic links at the bottom of the post, for those who want to play with the mechanics, search for themselves and others, etc.

Re:Geekiest Hacker News comments from the last month by eitally on Monday August 23, 2010 @04:15PM (#1627908)
as an addendum, since none of the results seemed overly technical to me, here are the "technical" words they used to gauge geekiness:

CS, Clojure, Debian, Haskell, JavaScript, Python, Rails, Scala, algorithm, compiler, engineer, frameworks, jQuery, macros, open-source, process, servers, stack

Re:Geekiest Hacker News comments from the last month by dantheman on Monday August 23, 2010 @04:38PM (#1627978)
Ahh, I see they left out Erlang.
Re:Geekiest Hacker News comments from the last month by Vivtek on Monday August 23, 2010 @09:11PM (#1628600)
Unfortunately, early design phases made the assumption that geekiness would max at 100. Erlang violated that assumption, being 117% geeky, causing the score calculation to dump core.

It was deemed safest simply to drop Erlang from the corpus.

Re:Geekiest Hacker News comments from the last month by riffer on Monday August 23, 2010 @04:50PM (#1628007)
Good point, in the case of Erlang it wasn't a picking favorites thing, so much as for whatever reason Erlang was not particularly well represented in the corpus of 25k comments that we grabbed from the site.
Re:Geekiest Hacker News comments from the last month by Vivtek on Monday August 23, 2010 @09:11PM (#1628604)
Honestly - if a particular technical topic is mentioned less that arguably makes it more geeky.
Re:Geekiest Hacker News comments from the last month by dman on Monday August 23, 2010 @05:04PM (#1628058)
On the topic of things being left out - does anyone have ideas on how to automate the process of finding seed terms for a given topic ?
Re:Geekiest Hacker News comments from the last month by Sukotto on Monday August 23, 2010 @11:09PM (#1628805)
Well, if I were doing it, I would search for words and short phrases that have their own Wikipedia pages. Give more weight to those whose pages contain a lot of text, have inline images, or have particularly contentious editing/reverting/meta-talk.

Likewise search for those same words and phrases in Google. Weight searches with fewer results more heavily.

Re:Geekiest Hacker News comments from the last month by dman on Monday August 23, 2010 @11:53PM (#1628864)
Thanks for the wikipedia suggestion - will try that one out. I am unclear about the terms of use for google search and if we can make use of search results in our tool.
Re:Geekiest Hacker News comments from the last month by _b8r0 on Tuesday August 24, 2010 @12:19PM (#1630220)
And ASM. All the cool kids write web apps in x86, except for the really cool 6502 kids in the corner.
Re:Geekiest Hacker News comments from the last month by tkhoven on Monday August 23, 2010 @09:14PM (#1628608)
And as for the "non-technical" words, was anyone else momentarily confused about the "less technical word like 'war'"? Or maybe I've been in java land for too long...
Re:Geekiest Hacker News comments from the last month by davidw on Tuesday August 24, 2010 @09:12AM (#1629648)
Definitely missing 'monad'. I completely understood the first comment in your list, but the monad stuff always loses me quickly.
Re:Geekiest Hacker News comments from the last month by Periodic on Monday August 23, 2010 @04:39PM (#1627981)
I also just noticed that they score capitalized and uncapitalized versions of their words differently.

For example, "scala" is 67.03/100, while "Scala" is 90.92/100.

Re:Geekiest Hacker News comments from the last month by riffer on Monday August 23, 2010 @04:56PM (#1628029)
Nice catch.

We generally keep the set of words case-sensitive because that can be helpful for disambiguation in broader corpuses ("Python" is often something different than "python"), and also because POS taggers tend to have trouble identifying proper nouns if case is not preserved).

But in this situation "scala" and "Scala" definitely should score similarly.

Re:Geekiest Hacker News comments from the last month by whimsy on Monday August 23, 2010 @09:53PM (#1628684)
I've never seen anyone mention a real python or ruby here. That doesn't mean it doesn't happen, but I think it's a safe assumption just to assume that the typist was lazy.
Re:Geekiest Hacker News comments from the last month by riffer on Tuesday August 24, 2010 @12:45AM (#1628996)
I didn't think it would ever happen either, and then I saw this: http://news.ycombinator.com/item?id=748632

On the essence, though, you're 100% right

Re:Geekiest Hacker News comments from the last month by jurjenh on Tuesday August 24, 2010 @04:20AM (#1629283)
I would go as far as to suggest that mentioning python in this manner (in this community) would in fact be geekier, thus merit a higher score.

Not sure about ruby, but would venture that it wouldn't make much of a difference on the scoring...

Re:Geekiest Hacker News comments from the last month by JacobAldridge on Monday August 23, 2010 @06:08PM (#1628226)
"But in this situation "scala" and "Scala" definitely should score similarly."

Not necessarily. I don't think it's a stretch to assume that those who capitalise Scala might be somewhat more geeky?

Re:Geekiest Hacker News comments from the last month by nl on Tuesday August 24, 2010 @12:36AM (#1628962)
I don't think it's a stretch to assume that those who capitalise Scala might be somewhat more geeky?

Scala should always start with a capital letter (it's a proper noun). People who use the lower case version either don't know, don't care, or are typing on a mobile keyboard.

Re:Geekiest Hacker News comments from the last month by JacobAldridge on Tuesday August 24, 2010 @02:50AM (#1629166)
"either don't know, don't care, or are typing on a mobile keyboard" and are therefore, I would contend, less 'geeky' than those of us who consistently apply the capital.

I mean, my username isn't particularly geeky, but enforcing the capitalisation must certainly be somewhere down that end of the spectrum!