News for hackers. Stuff that matters.
Re:Common Crawl by visarga on Friday March 26, 2021 @11:23PM (#26599640)
I recently did a pass through CC it took me 5 days, there were 80,000 1GB warc files. I had to just filter the information I wanted and dump the rest because I can't possibly host that much data on my machines. It's great data but very noisy, like the web is. Lots of repetition and useless junk mixed with useful data.

There are no comments on this comment yet.