Re:Show HN: Apifier – hosted web crawler for developers
by jacquesm on Tuesday October 20, 2015 @01:44PM
(#10420602)
Do you respect robots.txt? Do you publish your IP ranges?
[ Parent
| on HN ]
Re:Show HN: Apifier – hosted web crawler for developers
by jakubbalada on Tuesday October 20, 2015 @01:52PM
(#10420642)
Yes, by default we respect robots.txt. There is a switch to disable it - on your own responsibility. We don't fully respect Crawl-delay, but minimum delay between requests for all our crawlers is set to 2000ms. We don't publish our IP ranges yet.