Processing 40 TB of code from ~10M projects with a server and Go for $100 (2019)

Re:Processing 40 TB of code from ~10M projects with a server... by throwawaygog6 on Monday October 03, 2022 @06:04PM (#33074343)

>>> If someone wants to host the raw files to allow others to download it let me know. It is a 83 GB tar.gz file which uncompressed is just over 1 TB in size.

Anyone knows if it is possible to download similar data set for youtube and reddit? I have ideas for search engine based on it, but I don't want to write/maintain scraper scripts.

[ Parent | on HN ]

Re:Processing 40 TB of code from ~10M projects with a server... by teraflop on Monday October 03, 2022 @07:40PM (#33075086)

There's a very large dataset of Reddit posts and comments at https://files.pushshift.io/reddit/

[ Parent | on HN ]

Re:Processing 40 TB of code from ~10M projects with a server... by testesttest4 on Monday October 03, 2022 @06:31PM (#33074581)

I have most of the historical reddit data except ~year I use to train ML models. Let me see if I can find a public link for you...

[ Parent | on HN ]