Re:Who exactly is crawling my site?
by ceejayoz on Friday October 18, 2013 @03:17PM
(#6573412)
Sitemaps (http://www.sitemaps.org/) help a bit in that regard, as crawlers can check the sitemap and only crawl updated content.
[ Parent
| on HN ]
Re:Who exactly is crawling my site?
by unhappyhippie on Friday October 18, 2013 @05:21PM
(#6574088)
Thanks.