Re:Common Crawl
by gillesjacobs on Friday March 26, 2021 @02:42PM
(#26595427)
This is mainly used for language modeling research. A filtered CC was used in GPT3 and I have personally used data from CC for NLP projects.
[ Parent
| on HN ]
I recently did a pass through CC it took me 5 days, there were 80,000 1GB warc files. I had to just filter the information I wanted and dump the rest because I can't possibly host that much data on my machines. It's great data but very noisy, like the web is. Lots of repetition and useless junk mixed with useful data.