Re:Yandex can now search for Facebook posts
by mapleoin on Tuesday January 14, 2014 @11:01AM
(#7057691)
They give the Internet Archive the most access, but oddly go out of their way to block their TOS and privacy policy from them and not anyone else. Sneaky.
It's actually the opposite. They allow IA access to their terms and policy, but block everything else:
It's actually the opposite. They allow IA access to their terms and policy, but block everything else:
User-agent: ia_archiver
Allow: /about/privacy
Allow: /full_data_use_policy
Allow: /legal/terms
Allow: /policy.php
Disallow: /