If I'm remembering correctly, the raw data was given under an NDA/DND as a one time only deal. There was a subreddit associated with the data and collection but its since been banned.

I was the one with this data. The subreddit which discussed provided the data went private.

I have the raw data but it's infeasable to distribute due to size and the original source got in trouble for making it easily accessible.

Yeah the thing I most wanted out of the post is a torrent to the data.

I have been working on a small shell script to get the top 500 posts of the top 5000 subreddits. The big issue is the api call limit is 60 req/min for OAuth2 tokens. It would save a lot of time if someone made a torrent of a data set like that, I will probably post the data I collect somewhere as well but it would be easier if reddit just hosted a data set like the one I described. They could probably post one on AWS public data sets. https://aws.amazon.com/datasets/

Could you not just request the .json feed whenever you like without limits? ( by appending .json to the reddit URL )

I'm a little new to this but my understanding is that that would be the same as doing a GET request to the REST API. Which would have the same penalty for going over 30 req/s without an OAuth token.

For example, this is what my get request to the REST endpoint looks like without URL parameters:

GET reddit.com/r/explainlikeimfive/comments/2w1xzo.json

Which returns the same response as appending .json to the the URL:

https://www.reddit.com/r/explainlikeimfive/comments/2w1xzo/e...

Note that almost all subreddits only let you travel back 1,000 posts using the .json feed.

The exception is /r/all, which had infinite pagination, but the low API limits prevent much usability.

What kind of data are you pulling? Have you tried scraping the subreddit RSS feeds?

I haven't tried that, I am looking for comments attached to usernames attached to articles/other comments in a tree and the subreddit that they are contained within. In order to define each user as an agent who has a probabilistic membership to the set of all interest groups (subpopulations) within the data set defined by a mixture model.

My pipeline right now looks like:

list of most popular SFW subreddits of all time -> gnu parallel -> curl on comment listings of top articles within subreddit -> json processing with jq -> gzip -> store on filesystem

So at the end I should have a directory of 5000 subreddits containing a gzipped json file of the comment threads of their top 500 posts.

I'm new to this kind of data processing but this is the easiest way I have found to do a short term, one time scrape of this data.

The author released data, links are at the bottom of the article (great thanks to him!).

Thanks for the tip, I just started downloading the data.

The admins released a bunch of anonymize voting data once. And several people have distributed datasets of scraped comments. Sorry I don't have links handy. Check through /r/redditdev

Thanks for the tip, I found the link you're referring to:

https://github.com/umbrae/reddit-top-2.5-million