News for hackers. Stuff that matters.
Re:Yandex can now search for Facebook posts by jonknee on Tuesday January 14, 2014 @10:56AM (#7057653)
FWIW, here's Facebook's robots.txt file. While this was a firehose agreement and not subject to robots.txt, it is an interesting look:

https://facebook.com/robots.txt

They give the Internet Archive the most access, but oddly go out of their way to block their TOS and privacy policy from them and not anyone else. Sneaky.

Update: scratch that, I missed that they were Allow statements for ia_archiver. The Internet Archive has by far the least access.

Re:Yandex can now search for Facebook posts by mapleoin on Tuesday January 14, 2014 @11:01AM (#7057691)
They give the Internet Archive the most access, but oddly go out of their way to block their TOS and privacy policy from them and not anyone else. Sneaky.

It's actually the opposite. They allow IA access to their terms and policy, but block everything else:

User-agent: ia_archiver

Allow: /about/privacy

Allow: /full_data_use_policy

Allow: /legal/terms

Allow: /policy.php

Disallow: /

Re:Yandex can now search for Facebook posts by jonknee on Tuesday January 14, 2014 @11:03AM (#7057706)
Duh, all the others were Disallow statements and I just missed that the IA ones were Allow. Thanks for the tip.