News for hackers. Stuff that matters.

A comment on Tell HN: The front page of Hacker News has been deindexed from Google. Parent comment

Re:Tell HN: The front page of Hacker News has been deindexed... by chintan on Thursday June 27, 2013 @09:25PM (#5955712)
Doesn't Googlebot respect Crawl-Delay in robots.txt? PG has set it to be 30 secs - https://news.ycombinator.com/robots.txt - which IMHO should not cause any load issues given the overall traffic profile of HN.
[ Parent | on HN ]
Re:Tell HN: The front page of Hacker News has been deindexed... by kanamekun on Thursday June 27, 2013 @10:02PM (#5955782)
Googlebot doesn't respect the crawl-delay setting in robots.txt. https://groups.google.com/forum/#!topic/Google_Webmaster_Hel...

As I understand it, the best way to lower the crawl rate is to log into Google Webmaster Tools and manually lower your crawl rate. The crawl rate delays expire every 90 days, so I set a calendar reminder to renew the crawl delay every 3 months. https://support.google.com/webmasters/answer/48620?hl=en

[ Parent | on HN ]