News for hackers. Stuff that matters.
Re:The Evolution of Marginalia's Crawling by marginalia_nu on Tuesday August 23, 2022 @04:47PM (#32571260)
Yeah I stand by that.

I just don't see what Common Crawl would actually help me with, other than making my own data more stale given it would take about as long to download the CC dataset as my own crawl takes to perform (i.e. ~200h).

As it stands, crawling isn't the hard part of building a search engine. Don't get me wrong, if you're doing data science and want to access a crawl data set, Common Crawl is amazing.

There are no comments on this comment yet.