News for hackers. Stuff that matters.
Re:Processing 40 TB of code from ~10M projects with a server... by jwilk on Monday October 03, 2022 @04:40PM (#33073378)
Previously:

https://news.ycombinator.com/item?id=21121735 (80 comments)

[ Parent | on HN ]
Re:Processing 40 TB of code from ~10M projects with a server... by KolmogorovComp on Monday October 03, 2022 @04:01PM (#33072866)
Oops linked to an anchor in the body and cannot edit. Correct link: https://boyter.org/posts/an-informal-survey-of-10-million-gi...
[ Parent | on HN ]
Re:Processing 40 TB of code from ~10M projects with a server... by layer8 on Monday October 03, 2022 @06:48PM (#33074711)
Average number of Makefiles is 1518.1769808494607? That can’t be right?
[ Parent | on HN ]
Re:Processing 40 TB of code from ~10M projects with a server... by CamJN on Monday October 03, 2022 @07:12PM (#33074902)
The ”average person eats 3 spiders a year” factoid is actually just statistical error. The average person eats 0 spiders per year. Spiders Georg, who lives in a cave & eats over 10,000 each day, is an outlier and should not have been counted”
[ Parent | on HN ]
Re:Processing 40 TB of code from ~10M projects with a server... by layer8 on Monday October 03, 2022 @08:04PM (#33075251)
Of course, however given that there are 1.2 million repositories with makefiles, the listed average would mean that they contain 1.8 billion makefiles in total. I doubt that that’s the case.

Also, according to the page, only 59 million of all files are named “Makefile”.

I suspect that the file language recognition has major false positives for the Makefile language.

[ Parent | on HN ]
Re:Processing 40 TB of code from ~10M projects with a server... by trevor-e on Monday October 03, 2022 @07:42PM (#33075098)
Yea this clearly seems like a bug or some repo skewing the results. Also a good reminder why the median should often be shown alongside average values.
[ Parent | on HN ]
Re:Processing 40 TB of code from ~10M projects with a server... by thethirdone on Monday October 03, 2022 @04:51PM (#33073514)
Does anyone know why "pawn" is considered an offensive word? It seems likely it ranks highly on the offensive word list because of chess engines.
[ Parent | on HN ]
Re:Processing 40 TB of code from ~10M projects with a server... by ok_dad on Monday October 03, 2022 @05:57PM (#33074287)
I also noticed "knob" and "xxx" near the top of that list (and others) for which I understand why they are considered "offensive" in some contexts, but in code those are quite non-offensive (unless, of course, the context is offensive, haha).
[ Parent | on HN ]
Re:Processing 40 TB of code from ~10M projects with a server... by yaddaor on Monday October 03, 2022 @05:41PM (#33074089)
Phonetically it sounds like "porn".
[ Parent | on HN ]
Re:Processing 40 TB of code from ~10M projects with a server... by throwawaygog6 on Monday October 03, 2022 @06:04PM (#33074343)
>>> If someone wants to host the raw files to allow others to download it let me know. It is a 83 GB tar.gz file which uncompressed is just over 1 TB in size.

Anyone knows if it is possible to download similar data set for youtube and reddit? I have ideas for search engine based on it, but I don't want to write/maintain scraper scripts.

[ Parent | on HN ]
Re:Processing 40 TB of code from ~10M projects with a server... by teraflop on Monday October 03, 2022 @07:40PM (#33075086)
There's a very large dataset of Reddit posts and comments at https://files.pushshift.io/reddit/
[ Parent | on HN ]
Re:Processing 40 TB of code from ~10M projects with a server... by testesttest4 on Monday October 03, 2022 @06:31PM (#33074581)
I have most of the historical reddit data except ~year I use to train ML models. Let me see if I can find a public link for you...
[ Parent | on HN ]
Re:Processing 40 TB of code from ~10M projects with a server... by layer8 on Monday October 03, 2022 @06:49PM (#33074720)
It would be interesting to plot some of the statistics by age of repository.
[ Parent | on HN ]