Robots.txt is a suicide note

Re:Robots.txt is a suicide note by tszming on Tuesday May 10, 2011 @02:12AM (#2531570)

Some parameter are useful, but they are not part of the standard.

e.g. Crawl-Delay, prevent DDOS from YAHOO! Slurp

[ Parent | on HN ]

Re:Robots.txt is a suicide note by soult on Tuesday May 10, 2011 @02:40AM (#2531628)

Is Yahoo! Slurp still active? I thought they turned it off when they switched to using Bing search results on Yahoo!

[ Parent | on HN ]

Re:Robots.txt is a suicide note by tszming on Tuesday May 10, 2011 @03:40AM (#2531742)

I don't think so. I still receive a lot of requests from Yahoo! Slurp / Yahoo! Slurp China this month.

[ Parent | on HN ]

Re:Robots.txt is a suicide note by yaix on Tuesday May 10, 2011 @04:51AM (#2531852)

I saw a few hits from Slurp with Y!/USA IP just 2 days ago in my logs.

[ Parent | on HN ]