News for hackers. Stuff that matters.

A comment on Here is the robots.txt of Google.

Re:Here is the robots.txt of Google by twoz on Wednesday July 08, 2009 @12:43PM (#694184)
Does Yahoo! not have a robots.txt file?

http://yahoo.com/robots.txt Sorry, the page you requested was not found.

[ Parent | on HN ]
Re:Here is the robots.txt of Google by jdrock on Wednesday July 08, 2009 @01:56PM (#694314)
If that page doesn't exist, then according to specification, they don't and you can crawl any page.
[ Parent | on HN ]
Re:Here is the robots.txt of Google by twoz on Wednesday July 08, 2009 @05:09PM (#694706)
Update: Some Yahoo! subdomains have 'em.

Yes:

    http://search.yahoo.com/robots.txt
    http://groups.yahoo.com/robots.txt
    http://realestate.yahoo.com/robots.txt
No:

    http://maps.yahoo.com/robots.txt
    http://omg.yahoo.com/robots.txt
[ Parent | on HN ]