News for hackers. Stuff that matters.

A comment on Robots.txt. Parent comment

Re:Robots.txt by signifiers on Monday February 11, 2013 @01:34PM (#5202362)
I bet they are fairly liberal with their size limits:

http://www.amazon.com/robots.txt http://aws.amazon.com/robots.txt http://www.google.com/robots.txt http://store.apple.com/robots.txt

[ Parent | on HN ]
Re:Robots.txt by martin-adams on Monday February 11, 2013 @02:35PM (#5202765)
Apple should use git instead of subversion :)
[ Parent | on HN ]