News for hackers. Stuff that matters.
Re:Tell HN: The front page of Hacker News has been deindexed... by kanamekun on Thursday June 27, 2013 @10:02PM (#5955782)
Googlebot doesn't respect the crawl-delay setting in robots.txt. https://groups.google.com/forum/#!topic/Google_Webmaster_Hel...

As I understand it, the best way to lower the crawl rate is to log into Google Webmaster Tools and manually lower your crawl rate. The crawl rate delays expire every 90 days, so I set a calendar reminder to renew the crawl delay every 3 months. https://support.google.com/webmasters/answer/48620?hl=en

There are no comments on this comment yet.