News for hackers. Stuff that matters.
Re:Ask HN: Why is Facebook indexed in search engine, against... by eel on Saturday September 14, 2013 @04:25PM (#6386948)
It's not against the robots.txt rules. In Facebook's robots.txt file, there is a section for the Googlebot (as well as several other bots)

  User-agent: Googlebot
  Disallow: /ac.php
  ...
A bot is supposed to follow the its own User-agent section. If that is not found, it should follow the User-agent: * section. [1]

[1] http://www.robotstxt.org/orig.html

Re:Ask HN: Why is Facebook indexed in search engine, against... by hiby007 on Saturday September 14, 2013 @04:46PM (#6387016)
Great Reply, thanks.
Re:Ask HN: Why is Facebook indexed in search engine, against... by techaddict009 on Saturday September 14, 2013 @04:27PM (#6386956)
Any supporting link reference for this ?
Re:Ask HN: Why is Facebook indexed in search engine, against... by eel on Saturday September 14, 2013 @04:28PM (#6386961)
I edited the original comment, but yes, it is in this document: http://www.robotstxt.org/orig.html

The key paragraph is:

  If the value is '*', the record describes the default
  access policy for any robot that has not matched any of
  the other records. It is not allowed to have multiple
  such records in the "/robots.txt" file.
Re:Ask HN: Why is Facebook indexed in search engine, against... by techaddict009 on Saturday September 14, 2013 @04:31PM (#6386970)
Thanks for sharing this !