News for hackers. Stuff that matters.
Re:Ask PG: Karma Data Dump by jacquesm on Monday September 14, 2009 @03:02PM (#822368)
For the life of me I can't find the sql connection in arc, would you help me find it ;) ?

http://ycombinator.com/arc/arc3.tar

Personally I think releasing that data is a bad idea. It might help people to game the system and it might inadvertently release more than intended, see:

http://arstechnica.com/tech-policy/news/2009/09/your-secrets...

Re:Ask PG: Karma Data Dump by chasingsparks on Monday September 14, 2009 @03:13PM (#822385)
I am not sure the game-the-system argument is strong. I could write a scraper to get the same data I am requesting in about 20 minutes. I cannot speak for other members, but I personally would want a karma dump to see if I can improve the algorithm. It is already pretty robust against abuses, but I think it could be improved as far as ranking goes.
Re:Ask PG: Karma Data Dump by jacquesm on Monday September 14, 2009 @03:33PM (#822427)
HN has pretty good bot blocking measures, a little while ago someone posted a less-then-optimal script to retrieve the users karma on the fly and almost all of the users of that script got blocked.

Datamining a body of data like that is likely to turn up all kinds of interesting tidbits, amongst others you could use it to identify the 30 or so 'secret' power users on HN, figure out when they're sleeping and so maximize your take on spammed links.

Even now it can take more than 2 hours before a spam link gets deleted, no need to make it worse.

Not everybody uses HN with good intentions, there are plenty of people that see this as just another place to dump their links.

Re:Ask PG: Karma Data Dump by zck on Monday September 14, 2009 @03:59PM (#822487)
If you're looking for power users, it's public who the 100 users with the most karma are: http://news.ycombinator.com/leaders
Re:Ask PG: Karma Data Dump by anigbrowl on Monday September 14, 2009 @04:36PM (#822561)
Indeed, but using that metric might lead you to conclude that pg and other luminaries are just bit players :)
Re:Ask PG: Karma Data Dump by nostrademons on Monday September 14, 2009 @05:20PM (#822644)
PG would be number 1 by almost a 2:1 margin if he were on the list (he has 50,000+ karma, vs. 27,000 for nickb). The software explicitly blocks him (and has since HN was released), presumably as a way to make the site feel like a community and not his personal vanity site.
Re:Ask PG: Karma Data Dump by zck on Monday September 14, 2009 @06:23PM (#822734)
That page doesn't include any admins. The page is defined on line 2258: (newsop leaders () (leaderspage user))

Leaderspage is just below: line 2262. The line we're interested in is line 2266: (each u (firstn nleaders* (leading-users)) This line says "for each of the first nleaders* in the list returned by (leading-users), do the following:".

Leading-users is line 2276. It returns a sorted list of users by karma. When looking at a user and deciding whether to sort em, it asks two questions: "does this user have more karma than the constant leader-threshold*?" and "is this user not an admin?"

So if you ever find a user who has more karma than whoever is the last user on the leaders page, but isn't on it, either you've found an admin, or the code running this site has been changed from the 3.1 release.

Re:Ask PG: Karma Data Dump by jacquesm on Monday September 14, 2009 @07:05PM (#822818)
There is only one user like that afaik.
Re:Ask PG: Karma Data Dump by kogir on Monday September 14, 2009 @03:31PM (#822417)
Why assume that SQL is involved at all? Hint: it's not.
Re:Ask PG: Karma Data Dump by jacquesm on Monday September 14, 2009 @03:34PM (#822432)
That was the point.
Re:Ask PG: Karma Data Dump by chasingsparks on Monday September 14, 2009 @03:35PM (#822433)
TSV, JSON, or whatever then -- it's the less relevant detail. I just want a dump and I consider it uncouth to scrape if someone is willing to freely provide a dump -- or at least, I would prefer permission.
Re:Ask PG: Karma Data Dump by mnemonik on Monday September 14, 2009 @03:48PM (#822463)
It is all flat files. Read the arc code, it is very interesting.
Re:Ask PG: Karma Data Dump by joubert on Monday September 14, 2009 @04:33PM (#822555)
Is it transactional?
Re:Ask PG: Karma Data Dump by wlievens on Tuesday September 15, 2009 @03:16AM (#823443)
I think it's the write-to-a-new-file-every-time kind of transactional. That probably works just fine for this kind of application.
Re:Ask PG: Karma Data Dump by joubert on Tuesday September 15, 2009 @11:40AM (#824038)
OK, so if your data model depends on multiple files for persistence, there's no transactional behavior?