Re:Facebook's robots.txt
by toomuchtodo on Friday November 29, 2013 @07:07PM
(#6821603)
You just ignore the robots.txt file, crawl slowly, and from distributed virtual machines.
Not that you should do that. Robots.txt is a nicety though, the client doesn't have to respect it, and the server doesn't have to allow your HTTP requests.
Not that you should do that. Robots.txt is a nicety though, the client doesn't have to respect it, and the server doesn't have to allow your HTTP requests.