News for hackers. Stuff that matters.
Re:Who exactly is crawling my site? by ceejayoz on Friday October 18, 2013 @03:17PM (#6573412)
Sitemaps (http://www.sitemaps.org/) help a bit in that regard, as crawlers can check the sitemap and only crawl updated content.
Re:Who exactly is crawling my site? by unhappyhippie on Friday October 18, 2013 @05:21PM (#6574088)
Thanks.