News for hackers. Stuff that matters.
Re:Ask HN: Why is Facebook indexed in search engine, against... by Scryptonite on Saturday September 14, 2013 @04:12PM (#6386915)
What if a different robots.txt is being served up for the real Googlebots?

EDIT:

Based on the comments in their robots.txt it appears that they are whitelisting certain robots. You would have to apply for your robot to crawl their site at https://www.facebook.com/apps/site_scraping_tos.php

They probably serve up unique generated robots.txt based on whitelisted robots. You'd never know what rules it contains unless you are a whitelisted robot.

Re:Ask HN: Why is Facebook indexed in search engine, against... by beneills on Saturday September 14, 2013 @04:24PM (#6386944)
Or your user-agent is the same as one. Unless they are whitelisting IPs.

I tested this with: wget --user-agent "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" http://www.facebook.com/robots.txt

and diffed, but the results are the same.

Re:Ask HN: Why is Facebook indexed in search engine, against... by phyalow on Saturday September 14, 2013 @04:27PM (#6386959)
Is your IP range the same as googles?
Re:Ask HN: Why is Facebook indexed in search engine, against... by tazzy531 on Saturday September 14, 2013 @04:33PM (#6386973)
Google doesn't release their IP ranges. It can come from any IP.