News for hackers. Stuff that matters.

A comment on Yahoo is Shutting Down Del.icio.us. Parent comment

Re:Yahoo is Shutting Down Del.icio.us by Tichy on Thursday December 16, 2010 @05:50PM (#2014193)
User-agent: * Disallow: /

I don't remember the robots.txt rules for sure, but doesn't that mean they don't allow crawlers at all?

[ Parent | on HN ]
Re:Yahoo is Shutting Down Del.icio.us by ivank on Thursday December 16, 2010 @06:17PM (#2014315)
That's the rule for crawlers that aren't Slurp, Googlebot, Teoma, or msnbot.
[ Parent | on HN ]
Re:Yahoo is Shutting Down Del.icio.us by Tichy on Thursday December 16, 2010 @06:40PM (#2014390)
I noticed the extra rules, but I am neither Slurp, Googlebot, Teoma nor msnbot :-(
[ Parent | on HN ]
Re:Yahoo is Shutting Down Del.icio.us by mfukar on Friday December 17, 2010 @08:29AM (#2016117)
robots.txt is merely a suggestion.
[ Parent | on HN ]