News for hackers. Stuff that matters.
Re:Facebook's robots.txt by perryh2 on Friday November 29, 2013 @06:35PM (#6821514)
Re:Facebook's robots.txt by glomph on Saturday November 30, 2013 @07:35AM (#6822970)
Re:Facebook's robots.txt by tux on Friday November 29, 2013 @08:01PM (#6821761)
Also has a funny error 404 when you remove "s"

Uh oh... Something didn't work. > http://disqus.com/human.txt

Re:Facebook's robots.txt by usaphp on Friday November 29, 2013 @08:44PM (#6821904)
What's "bmw" doing on the top of his head ?
Re:Facebook's robots.txt by viana007 on Friday November 29, 2013 @06:32PM (#6821504)
Re:Facebook's robots.txt by joshguthrie on Saturday November 30, 2013 @08:16AM (#6823056)
Chrome user here. When I open it, the tab is automatically closed.

Tried to curl it, exact content, no 302 towards a "<script>window.close</script>",... Got anything?

Re:Facebook's robots.txt by easy_rider on Friday November 29, 2013 @07:36PM (#6821679)
/* would have sufficed
Re:Facebook's robots.txt by darkmighty on Friday November 29, 2013 @08:08PM (#6821786)
Re:Facebook's robots.txt by kr1m on Friday November 29, 2013 @05:48PM (#6821370)
You don't scrape Facebook, Facebook scrapes you!
Re:Facebook's robots.txt by jgalt212 on Friday November 29, 2013 @07:27PM (#6821656)
In the US, you catch a cold. In Soviet Russia, cold catches you!

http://en.wikipedia.org/wiki/Russian_reversal

Re:Facebook's robots.txt by yalogin on Friday November 29, 2013 @07:10PM (#6821608)
So what does it mean by facebook whitelisting a scraping service? Do they actively block scrapers?
Re:Facebook's robots.txt by dblacc on Friday November 29, 2013 @07:22PM (#6821643)
I could be wrong but I believe that the the default is that spiders are blocked and only the "User-Agents" listed are allowed to scrape (but not the disallow pages).
Re:Facebook's robots.txt by elbear on Friday November 29, 2013 @08:28PM (#6821866)
You are correct.
Re:Facebook's robots.txt by pdfcollect on Friday November 29, 2013 @05:52PM (#6821378)
Is there a way to replace this robots.txt with a null robots.txt? :)
Re:Facebook's robots.txt by toomuchtodo on Friday November 29, 2013 @07:07PM (#6821603)
You just ignore the robots.txt file, crawl slowly, and from distributed virtual machines.

Not that you should do that. Robots.txt is a nicety though, the client doesn't have to respect it, and the server doesn't have to allow your HTTP requests.

Re:Facebook's robots.txt by bibstha on Saturday November 30, 2013 @01:05AM (#6822452)
What is a User Agent: Yeti?
Re:Facebook's robots.txt by unfunco on Saturday November 30, 2013 @05:50AM (#6822826)
It's the crawler for Naver, a south Korean search engine.
Re:Facebook's robots.txt by decasteve on Friday November 29, 2013 @07:06PM (#6821597)
Even Facebook's robots.txt has a hatred for my pseudo-anonymous browser settings. Facebook gives me this (for any page): "Sorry, something went wrong. We're working on getting this fixed as soon as we can."
Re:Facebook's robots.txt by startling on Friday November 29, 2013 @07:11PM (#6821610)
robots.txt isn't enforced.
Re:Facebook's robots.txt by easy_rider on Friday November 29, 2013 @07:40PM (#6821686)
Maybe they should be. Gentleman's agreements do not apply to robots.
Re:Facebook's robots.txt by cheald on Friday November 29, 2013 @09:28PM (#6822022)
And how exactly do you propose verifying that the user agent purporting to be Googlebot or Firefox is actually who they are? They're inherently unenforceable.

robots.txt is basically a list of rules that lay out "This is how we'd like you to crawl us. We might stop serving you if you don't comply", rather than a hard-and-fast set of directives that specify how a webcrawler will be guaranteed to behave.

Re:Facebook's robots.txt by easy_rider on Saturday November 30, 2013 @11:16AM (#6823499)
You can implement some strict enforcing in Apache using some crafty mod_rewrite stuff: http://andthatsjazz.org/defeat.html

User-agent is to easily spoofed, but we could check if the robots are indeed Google (whitelisted) and not some other crawler that just wants to scrape your content.

In the realm of mail servers we have something called SPF: http://en.wikipedia.org/wiki/Sender_Policy_Framework

Just thinking out of the box here, but other than checking IP ranges: Maybe a hash being sent as a header inside the GET request by the crawler to verify if they are who they say they are.