Re:Facebook's robots.txt
by yalogin on Friday November 29, 2013 @07:10PM
(#6821608)
So what does it mean by facebook whitelisting a scraping service? Do they actively block scrapers?
[ Parent
| on HN ]
I could be wrong but I believe that the the default is that spiders are blocked and only the "User-Agents" listed are allowed to scrape (but not the disallow pages).