Re:Ask HN: Why is Facebook indexed in search engine, against...
by techaddict009 on Saturday September 14, 2013 @04:27PM
(#6386956)
Any supporting link reference for this ?
[ Parent
| on HN ]
Re:Ask HN: Why is Facebook indexed in search engine, against...
by eel on Saturday September 14, 2013 @04:28PM
(#6386961)
I edited the original comment, but yes, it is in this document: http://www.robotstxt.org/orig.html
The key paragraph is:
If the value is '*', the record describes the default
access policy for any robot that has not matched any of
the other records. It is not allowed to have multiple
such records in the "/robots.txt" file.
Re:Ask HN: Why is Facebook indexed in search engine, against...
by techaddict009 on Saturday September 14, 2013 @04:31PM
(#6386970)
Thanks for sharing this !