Re:Ask HN: Why is Facebook indexed in search engine, against...
by hiby007 on Saturday September 14, 2013 @04:46PM
(#6387016)
Great Reply, thanks.
Re:Ask HN: Why is Facebook indexed in search engine, against...
by techaddict009 on Saturday September 14, 2013 @04:27PM
(#6386956)
Any supporting link reference for this ?
Re:Ask HN: Why is Facebook indexed in search engine, against...
by eel on Saturday September 14, 2013 @04:28PM
(#6386961)
I edited the original comment, but yes, it is in this document: http://www.robotstxt.org/orig.html
The key paragraph is:
If the value is '*', the record describes the default
access policy for any robot that has not matched any of
the other records. It is not allowed to have multiple
such records in the "/robots.txt" file.
Re:Ask HN: Why is Facebook indexed in search engine, against...
by techaddict009 on Saturday September 14, 2013 @04:31PM
(#6386970)
Thanks for sharing this !
[1] http://www.robotstxt.org/orig.html