News for hackers. Stuff that matters.
Re:Ask HN: Why is Facebook indexed in search engine, against... by techaddict009 on Saturday September 14, 2013 @04:27PM (#6386956)
Any supporting link reference for this ?
Re:Ask HN: Why is Facebook indexed in search engine, against... by eel on Saturday September 14, 2013 @04:28PM (#6386961)
I edited the original comment, but yes, it is in this document: http://www.robotstxt.org/orig.html

The key paragraph is:

  If the value is '*', the record describes the default
  access policy for any robot that has not matched any of
  the other records. It is not allowed to have multiple
  such records in the "/robots.txt" file.
Re:Ask HN: Why is Facebook indexed in search engine, against... by techaddict009 on Saturday September 14, 2013 @04:31PM (#6386970)
Thanks for sharing this !