News for hackers. Stuff that matters.

A comment on Robots.txt. Parent comment

Re:Robots.txt by zalew on Monday February 11, 2013 @01:32PM (#5202351)
robots this size aren't that unusual http://www.bbc.co.uk/robots.txt
[ Parent | on HN ]
Re:Robots.txt by thenextcorner on Monday February 11, 2013 @03:28PM (#5203008)
I think you just proved the point here: SEO can be very valuable. Any good SEO would point out to the BBC that having similar rules for User-agent: Googlebot-Mobile and User-agent: * is not very efficient.
[ Parent | on HN ]
Re:Robots.txt by chayesfss on Monday February 11, 2013 @01:35PM (#5202368)
wow, just realized this is a great way to find information about websites
[ Parent | on HN ]