Re:Common Crawl
by visarga on Friday March 26, 2021 @11:23PM
(#26599640)
I recently did a pass through CC it took me 5 days, there were 80,000 1GB warc files. I had to just filter the information I wanted and dump the rest because I can't possibly host that much data on my machines. It's great data but very noisy, like the web is. Lots of repetition and useless junk mixed with useful data.
[ Parent
| on HN ]
There are no comments on this comment yet.