News for hackers. Stuff that matters.
Re:Full Hacker News database for download (posts, comments, ... by Anon84 on Sunday November 07, 2010 @10:56PM (#1881312)
I meant the actual YC/PG Hacker News database, not one reverse engineered from the HTML output.
Re:Full Hacker News database for download (posts, comments, ... by aditya on Sunday November 07, 2010 @10:59PM (#1881322)
What's the difference? Other than parsing issues?
Re:Full Hacker News database for download (posts, comments, ... by Anon84 on Sunday November 07, 2010 @11:03PM (#1881331)
Take a look at the example in the page:

    <HackerNews>
    <row>
        <ID>1</ID> 
        <Url>http://ycombinator.com</Url> 
        <Title>Y Combinator</Title> 
        <Text /> 
        <Username>pg</Username> 
        <Points>39</Points> 
        <Type>1</Type> 
        <Timestamp>2006-10-09T20:33:12.700</Timestamp> 
        <CommentCount>15</CommentCount> 
    </row>
    </HackerNews>
If you really want to understand the HN community (or online communities in general) this data doesn't tell you much. There are many interesting questions this data cannot answer.

When was each of those 39 points gained?

Who upvoted?

How many points did the submission have when each comment was posted?

How does the number of points affect the number of comments?

Are there cascades of votes (several users voting in quick succession)?

For each of the comments, who voted up and who voted down?

Re:Full Hacker News database for download (posts, comments, ... by mahmud on Sunday November 07, 2010 @11:37PM (#1881400)
I don't think the actual HN "database" has that much info. You would need to reconstruct this from the complete server logs, along with application logs for PG's direct updates (assuming he doesn't use the web interface)
Re:Full Hacker News database for download (posts, comments, ... by michaelhart on Monday November 08, 2010 @12:22AM (#1881498)
I imagine it does... It has to at least know who upvoted, to prevent revotes. The timestamp is frequently stored as well.
Re:Full Hacker News database for download (posts, comments, ... by Zev on Monday November 08, 2010 @12:30AM (#1881515)
It has to at least know who upvoted, to prevent revotes.

HN has to know who voted, not which way someone voted. HN doesn't let you change your vote once its been made, but it does let you downvote once you have enough karma.

Re:Full Hacker News database for download (posts, comments, ... by catch23 on Monday November 08, 2010 @01:32AM (#1881626)
Maybe they use a bloom filter to store who voted per comment, that would be an efficient way to prevent upvotes without needing to store much data.
Re:Full Hacker News database for download (posts, comments, ... by nkurz on Monday November 08, 2010 @03:45AM (#1881859)
I don't think this would work well. The occasional false-positives would look like annoying glitches, every now and then making it look like your account had been hijacked by hiding the buttons as if you had already 'voted'. Although maybe I'm not getting the math right?

Also, the existence of the 'saved stories' link from the user page would seem to indicate that for stories at least a full list is being stored.

Re:Full Hacker News database for download (posts, comments, ... by erikano on Monday November 08, 2010 @02:21AM (#1881726)
I think it would have been neat if someone created a browser-extension that let users contribute such data, but I guess preventing abuse could prove to be an impossible task, since there is no way that I can think of for a third-party to check whether the user really performed an action without having to perform those actions for the user, in which case we would have to trust our login-details to that third-party.
Re:Full Hacker News database for download (posts, comments, ... by Zev on Sunday November 07, 2010 @11:04PM (#1881335)
Timestamps and an accurate karma count (instead of the -4 cap) spring to mind. I'm sure there are other issues as well.