News for hackers. Stuff that matters.

A comment on Robots.txt. Parent comment

Re:Robots.txt by dsl on Monday February 11, 2013 @01:54PM (#5202510)
500kb is the hard upper limit in the crawler itself. Limits that are exposed elsewhere (Webmaster Tools, etc) are irrelevant.
[ Parent | on HN ]

There are no comments on this comment yet.