News for hackers. Stuff that matters.

A comment on Imdb-pie – Python IMDB client using the IMDB json web service for their iOS app.

Re:Imdb-pie – Python IMDB client using the IMDB json web ser... by berendhh on Monday January 21, 2013 @09:41AM (#5091639)
One Question: What robots.txt do you have to obey? http://app.imdb.com/robots.txt or http://www.imdb.com/robots.txt?

The second one is rather restrictive (and the first one does not exist...)

[ Parent | on HN ]
Re:Imdb-pie – Python IMDB client using the IMDB json web ser... by erikano on Monday January 21, 2013 @11:15AM (#5092062)
The Google crawlers, at least, will only consider a robots.txt on a subdomain valid for the subdomain it was served from.[1]

[1]: https://developers.google.com/webmasters/control-crawl-index...

[ Parent | on HN ]