What interests Reddit? A network analysis of 84M comments by 200K users

Re:What interests Reddit? A network analysis of 84M comments... by placeybordeaux on Wednesday February 18, 2015 @10:17AM (#9068900)

Yeah the thing I most wanted out of the post is a torrent to the data.

[ Parent | on HN ]

Re:What interests Reddit? A network analysis of 84M comments... by faizshah on Wednesday February 18, 2015 @10:26AM (#9068950)

I have been working on a small shell script to get the top 500 posts of the top 5000 subreddits. The big issue is the api call limit is 60 req/min for OAuth2 tokens. It would save a lot of time if someone made a torrent of a data set like that, I will probably post the data I collect somewhere as well but it would be easier if reddit just hosted a data set like the one I described. They could probably post one on AWS public data sets. https://aws.amazon.com/datasets/

[ Parent | on HN ]

Re:What interests Reddit? A network analysis of 84M comments... by HarryN on Wednesday February 18, 2015 @01:37PM (#9070123)

Could you not just request the .json feed whenever you like without limits? ( by appending .json to the reddit URL )

[ Parent | on HN ]

Re:What interests Reddit? A network analysis of 84M comments... by faizshah on Wednesday February 18, 2015 @02:38PM (#9070501)

I'm a little new to this but my understanding is that that would be the same as doing a GET request to the REST API. Which would have the same penalty for going over 30 req/s without an OAuth token.

For example, this is what my get request to the REST endpoint looks like without URL parameters:

GET reddit.com/r/explainlikeimfive/comments/2w1xzo.json

Which returns the same response as appending .json to the the URL:

https://www.reddit.com/r/explainlikeimfive/comments/2w1xzo/e...

[ Parent | on HN ]

Re:What interests Reddit? A network analysis of 84M comments... by minimaxir on Wednesday February 18, 2015 @01:53PM (#9070242)

Note that almost all subreddits only let you travel back 1,000 posts using the .json feed.

The exception is /r/all, which had infinite pagination, but the low API limits prevent much usability.

[ Parent | on HN ]

Re:What interests Reddit? A network analysis of 84M comments... by bduerst on Wednesday February 18, 2015 @12:32PM (#9069741)

What kind of data are you pulling? Have you tried scraping the subreddit RSS feeds?

[ Parent | on HN ]

Re:What interests Reddit? A network analysis of 84M comments... by faizshah on Wednesday February 18, 2015 @02:24PM (#9070431)

I haven't tried that, I am looking for comments attached to usernames attached to articles/other comments in a tree and the subreddit that they are contained within. In order to define each user as an agent who has a probabilistic membership to the set of all interest groups (subpopulations) within the data set defined by a mixture model.

My pipeline right now looks like:

list of most popular SFW subreddits of all time -> gnu parallel -> curl on comment listings of top articles within subreddit -> json processing with jq -> gzip -> store on filesystem

So at the end I should have a directory of 5000 subreddits containing a gzipped json file of the comment threads of their top 500 posts.

I'm new to this kind of data processing but this is the easiest way I have found to do a short term, one time scrape of this data.

[ Parent | on HN ]