Re:Who exactly is crawling my site?
by gizmo686 on Friday October 18, 2013 @05:40PM
(#6574193)
You could probably do this by having a robots.txt file that blocks all crawling. When you want to be re-crawled, you can edit it to allow the relevant spiders to crawl you. I would imagine that spiders do not automatically re-check sites which disallow everything often, but Google (and probably others) let you manually submit URLs to be crawled.
[ Parent
| on HN ]