News for hackers. Stuff that matters.

A comment on Robots.txt. Parent comment

Re:Robots.txt by kanzure on Monday February 11, 2013 @01:48PM (#5202465)
The robots.txt limit in Google's Webmasterpanelthing is 100,000 characters. Googlebot has been said to stop parsing robots.txt after 500,000 characters. I don't know if that's bytes or characters.
[ Parent | on HN ]
Re:Robots.txt by dsl on Monday February 11, 2013 @01:54PM (#5202510)
500kb is the hard upper limit in the crawler itself. Limits that are exposed elsewhere (Webmaster Tools, etc) are irrelevant.
[ Parent | on HN ]