Ask HN: Why is Facebook indexed in search engine, against robots.txt rules?
Facebooks robots.txt : https://www.facebook.com/robots.txt
Facebook Has several different rules for different search engines.
But if you go to the end of file. You will find this one rule.
User-agent: * Disallow: /
The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.
site:facebook.com returns About 3,520,000,000 results in google.
Then my question is why is facebook.com indexed in google and other search engines?
(
12 comments
| 11 points
| on HN
)
[1] http://www.robotstxt.org/orig.html
The key paragraph is:
EDIT:
Based on the comments in their robots.txt it appears that they are whitelisting certain robots. You would have to apply for your robot to crawl their site at https://www.facebook.com/apps/site_scraping_tos.php
They probably serve up unique generated robots.txt based on whitelisted robots. You'd never know what rules it contains unless you are a whitelisted robot.
I tested this with: wget --user-agent "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" http://www.facebook.com/robots.txt
and diffed, but the results are the same.
`User-agent: *` means for all other bots that didn't match any section above.
Here are the rules for Google:
- meaning anything else allowed.Yeah, Facebook's robots.txt file white-lists specific search engines and offer an option to get whitelisted ,then there is a wildcard disallow for everyone else.
Of course nefarious scrapers can ignore the robots.txt file or even spoof google or bingbot, but it least it sets a precedent and a policy that they can take further action on if needed.