News for hackers. Stuff that matters.
Re:Crawling Billions of Pages: Building Large Scale Crawling... by HaseebR7 on Friday June 12, 2015 @06:02AM (#9704904)
That is why you have sitemaps, for new websites with no sitemaps we could start with a default frequency, say, once a day. If it changes more often than that you can increase it to multiple times a day and so on. We can also look for the type of website to determine a starting frequency — for example an e-commerce websites ought to be crawled more frequently because new products, reviews, ratings could be added every few minutes whereas a personal blog may stay the same for weeks altogether.

There are no comments on this comment yet.