News for hackers. Stuff that matters.
Re:Common Crawl by gillesjacobs on Friday March 26, 2021 @02:42PM (#26595427)
This is mainly used for language modeling research. A filtered CC was used in GPT3 and I have personally used data from CC for NLP projects.
Re:Common Crawl by visarga on Friday March 26, 2021 @11:23PM (#26599640)
I recently did a pass through CC it took me 5 days, there were 80,000 1GB warc files. I had to just filter the information I wanted and dump the rest because I can't possibly host that much data on my machines. It's great data but very noisy, like the web is. Lots of repetition and useless junk mixed with useful data.