Re:Show HN: Apifier – hosted web crawler for developers
by jacquesm on Tuesday October 20, 2015 @01:44PM
(#10420602)
Do you respect robots.txt? Do you publish your IP ranges?
Re:Show HN: Apifier – hosted web crawler for developers
by jakubbalada on Tuesday October 20, 2015 @01:52PM
(#10420642)
Yes, by default we respect robots.txt. There is a switch to disable it - on your own responsibility. We don't fully respect Crawl-delay, but minimum delay between requests for all our crawlers is set to 2000ms. We don't publish our IP ranges yet.
Please have a look at the service, play with the examples and maybe set up your own crawl. My co-founder jakubbalada and myself will be around here to answer your questions. We'd love to hear what you guys think!