Re:Facebook's robots.txt
by dblacc on Friday November 29, 2013 @07:22PM
(#6821643)
I could be wrong but I believe that the the default is that spiders are blocked and only the "User-Agents" listed are allowed to scrape (but not the disallow pages).
[ Parent
| on HN ]