Re:The Evolution of Marginalia's Crawling
by marginalia_nu on Tuesday August 23, 2022 @04:47PM
(#32571260)
Yeah I stand by that.
I just don't see what Common Crawl would actually help me with, other than making my own data more stale given it would take about as long to download the CC dataset as my own crawl takes to perform (i.e. ~200h).
As it stands, crawling isn't the hard part of building a search engine. Don't get me wrong, if you're doing data science and want to access a crawl data set, Common Crawl is amazing.
https://hn.algolia.com/?query=author%3Amarginalia_nu%20commo... > https://news.ycombinator.com/item?id=32205535#32211292
> It's simply too unwieldy. It's far easier (and cheaper) to do my own crawling at a manageable scale, than it is to work with CC's datasets.
Is there any way to contribute to Common Crawl beyond donating?
https://commoncrawl.org/big-picture/what-you-can-do/