Re:Ask HN: Why is Facebook indexed in search engine, against...
by fendmark on Saturday September 14, 2013 @04:36PM
(#6386982)
Interesting
Yeah, Facebook's robots.txt file white-lists specific search engines and offer an option to get whitelisted
,then there is a wildcard disallow for everyone else.
Of course nefarious scrapers can ignore the robots.txt file or even spoof google or bingbot, but it least it sets a precedent and a policy that they can take further action on if needed.
Yeah, Facebook's robots.txt file white-lists specific search engines and offer an option to get whitelisted ,then there is a wildcard disallow for everyone else.
Of course nefarious scrapers can ignore the robots.txt file or even spoof google or bingbot, but it least it sets a precedent and a policy that they can take further action on if needed.