News for hackers. Stuff that matters.
Re:Facebook's robots.txt by yalogin on Friday November 29, 2013 @07:10PM (#6821608)
So what does it mean by facebook whitelisting a scraping service? Do they actively block scrapers?
Re:Facebook's robots.txt by dblacc on Friday November 29, 2013 @07:22PM (#6821643)
I could be wrong but I believe that the the default is that spiders are blocked and only the "User-Agents" listed are allowed to scrape (but not the disallow pages).
Re:Facebook's robots.txt by elbear on Friday November 29, 2013 @08:28PM (#6821866)
You are correct.