News for hackers. Stuff that matters.
Re:Hacker News bans Google and all other search engines by snprbob86 on Tuesday March 16, 2010 @12:42AM (#1195020)
I'd be really worried about some algorithm somewhere deciding to crawl less frequently even after the temporary ban has been lifted.
Re:Hacker News bans Google and all other search engines by jsrfded on Tuesday March 16, 2010 @01:01AM (#1195057)
Robot bans can render your archive.org records inaccessable, zap all your former code 200 urls out of the major search indices, and yes, kill your inclusion in the list of topical sites.

Much better to set a temporary Crawl-Delay directive. Otherwise you're not just telling the engines to pause crawling you, you're telling them "take all of my pages out of your index."

Re:Hacker News bans Google and all other search engines by Andys on Tuesday March 16, 2010 @01:29AM (#1195101)
What are the details of crawl-delay? If its supposed to be "number of seconds between hits", for some sites, 1 hit per second may as well be "not crawled at all"..
Re:Hacker News bans Google and all other search engines by ComputerGuru on Tuesday March 16, 2010 @01:43AM (#1195124)
Right, but his point is that at least this way you won't get delisted :)