News for hackers. Stuff that matters.
Re:Who exactly is crawling my site? by gizmo686 on Friday October 18, 2013 @05:40PM (#6574193)
You could probably do this by having a robots.txt file that blocks all crawling. When you want to be re-crawled, you can edit it to allow the relevant spiders to crawl you. I would imagine that spiders do not automatically re-check sites which disallow everything often, but Google (and probably others) let you manually submit URLs to be crawled.
Re:Who exactly is crawling my site? by walshemj on Friday October 18, 2013 @07:13PM (#6574577)
But a disallow all robots.txt will kill your sites traffic - its not going to work like you think it is.