topomorph

Coincidentally, I was also working on an HN Trends app a couple weeks ago (though I got distracted and haven't finished), and mine is normalized (by number of comments). For example: http://hn-trends.heroku.com/trends?q=scala%2Cclojure

Your app looks a lot better, though! (And my dataset ends around Sept. of last year.)

In case anyone wants to make their own trends (less sophisticated, but maybe more transparent), I have an app at http://hn-trends.heroku.com that plots word percentages on HN over time.

For example, some emotive words that appear to have increased since the beginning:

* sad: http://hn-trends.heroku.com/trends?q=sad

* fuck: http://hn-trends.heroku.com/trends?q=fuck

* shit: http://hn-trends.heroku.com/trends?q=shit

Some words that appear to have decreased:

* great: http://hn-trends.heroku.com/trends?q=great

* cool: http://hn-trends.heroku.com/trends?q=cool

* lol: http://hn-trends.heroku.com/trends?q=lol

* reddit: http://hn-trends.heroku.com/trends?q=reddit

* digg: http://hn-trends.heroku.com/trends?q=digg

I haven't looked at any of this closely, so make of it what you will.

(I was meaning to wait until I had time to dig deeper and add extra features like time series smoothers and trend/slope metrics before "releasing" this, but figured this might be useful now given the post. I still plan on adding those later.)

Just to be clear, that's the number of users with at least one submission in a given month. (Not the number of users with at least one submission or comment, as I sleepily wrote originally, and definitely not the number of distinct visitors.)

I used R's awesome ggplot2 package.

D'oh, Apple always strangely slips my mind when I think of software companies. I added a graph (http://dl.dropbox.com/u/10506/blog/hn-analysis/pct_apple.png). Interestingly, it's the company with the clearest upward trend.

Yeah, that measure was super hacky and I have no justification for it (but it was the simplest/only thing I could think of at the time). Do you have any suggestions?

One thing I'm thinking of now is seeing how (# points at the xth percentile) / (# points of 10th highest rated post) changes over time. (Or maybe I should just take a closer look at how the overall distribution of points changes to get a better feel first.)

SEEKING WORK - Remote preferred

(Buzzword: Data Scientist)

I'm primarily interested in freelance work, but full-time is also a possibility, and I'm willing to relocate. I'm currently in the Bay Area.

I have several years of experience in data mining, statistics, and machine learning (doing everything from quantitative trading and pricing, to building recommendation engines and mining search engines). I pick up new languages and technologies quickly, though I work primarily with Ruby, Python, Java, R, Rails, and Hadoop. To use a buzzword, you can think of me as the prototypical 'data scientist' -- I love doing everything from building dashboards to display data (Rails, Sinatra, Javascript), gathering data and cleaning it up (Ruby, Hadoop), exploring and analyzing data for patterns (R, Python), and coding machine learning algorithms for production (Java).

Blog, contact, and more information here: http://bit.ly/iP3aeU