Re:Hacker News bans Google and all other search engines
by Andys on Tuesday March 16, 2010 @01:29AM
(#1195101)
What are the details of crawl-delay? If its supposed to be "number of seconds between hits", for some sites, 1 hit per second may as well be "not crawled at all"..
Re:Hacker News bans Google and all other search engines
by ComputerGuru on Tuesday March 16, 2010 @01:43AM
(#1195124)
Right, but his point is that at least this way you won't get delisted :)
Much better to set a temporary Crawl-Delay directive. Otherwise you're not just telling the engines to pause crawling you, you're telling them "take all of my pages out of your index."