News for hackers. Stuff that matters.
Re:Ask HN: Why is Facebook indexed in search engine, against... by fendmark on Saturday September 14, 2013 @04:36PM (#6386982)
Interesting

Yeah, Facebook's robots.txt file white-lists specific search engines and offer an option to get whitelisted ,then there is a wildcard disallow for everyone else.

Of course nefarious scrapers can ignore the robots.txt file or even spoof google or bingbot, but it least it sets a precedent and a policy that they can take further action on if needed.

There are no comments on this comment yet.