Re:Who exactly is crawling my site?
by lostlogin on Friday October 18, 2013 @05:12PM
(#6574045)
This seems logical, however what about the person who updates rarely (once or twice a month) but get a lot of notice when they do. Search engines wouldn't want to miss the opportunity to get that traffic.
I wonder if there is some sort of reverse process you could opt into - ie never crawl me until I ask to be crawled.
Disclaimer: I know little about this sort of thing.
[ Parent
| on HN ]
You could probably do this by having a robots.txt file that blocks all crawling. When you want to be re-crawled, you can edit it to allow the relevant spiders to crawl you. I would imagine that spiders do not automatically re-check sites which disallow everything often, but Google (and probably others) let you manually submit URLs to be crawled.
But a disallow all robots.txt will kill your sites traffic - its not going to work like you think it is.
Re:Who exactly is crawling my site?
by pavel_lishin on Friday October 18, 2013 @07:17PM
(#6574584)
Presumably you could set a minimum interval - an hour, or a day. This would make sure that big updates are noticed, without putting undue stress on the server.