Re:What one may find in robots.txt
by realusername on Monday May 18, 2015 @09:31AM
(#9564277)
Robots.txt is supposed to be public. Its purpose is to recommend not to visit some links (it's only a recommendation). And he did not visit the link itself, he just read the comments (made for humans to read) in a public file, I don't think there is a problem here.
[ Parent
| on HN ]
There are no comments on this comment yet.