News for hackers. Stuff that matters.
Re:Who exactly is crawling my site? by pavel_lishin on Friday October 18, 2013 @03:19PM (#6573432)
Why not adjust the crawl rate based on how often the site's content changes? If DuckDuckGoogle crawls me once an hour for a week, and notices that I only updated my content twice, why not scale the crawling back to a more reasonable rate, like once a day?
Re:Who exactly is crawling my site? by lostlogin on Friday October 18, 2013 @05:12PM (#6574045)
This seems logical, however what about the person who updates rarely (once or twice a month) but get a lot of notice when they do. Search engines wouldn't want to miss the opportunity to get that traffic. I wonder if there is some sort of reverse process you could opt into - ie never crawl me until I ask to be crawled. Disclaimer: I know little about this sort of thing.
Re:Who exactly is crawling my site? by gizmo686 on Friday October 18, 2013 @05:40PM (#6574193)
You could probably do this by having a robots.txt file that blocks all crawling. When you want to be re-crawled, you can edit it to allow the relevant spiders to crawl you. I would imagine that spiders do not automatically re-check sites which disallow everything often, but Google (and probably others) let you manually submit URLs to be crawled.
Re:Who exactly is crawling my site? by walshemj on Friday October 18, 2013 @07:13PM (#6574577)
But a disallow all robots.txt will kill your sites traffic - its not going to work like you think it is.
Re:Who exactly is crawling my site? by pavel_lishin on Friday October 18, 2013 @07:17PM (#6574584)
Presumably you could set a minimum interval - an hour, or a day. This would make sure that big updates are noticed, without putting undue stress on the server.