News for hackers. Stuff that matters.
Re:Show HN: Apifier – hosted web crawler for developers by jancurn on Tuesday October 20, 2015 @12:58PM (#10420338)
Hello HN! Today we’re launching what we were building for the past couple of months. Apifier is a hosted web crawler for developers that enables them to extract data from any website using a few simple lines of JavaScript. We built it because we realized that many existing web scrapers trade off their ability to scrape complex websites for the "simplicity" of their user interface. We thought: we are programmers and we already use JavaScript for client-side development, so why not use it for scraping?

Please have a look at the service, play with the examples and maybe set up your own crawl. My co-founder jakubbalada and myself will be around here to answer your questions. We'd love to hear what you guys think!

Re:Show HN: Apifier – hosted web crawler for developers by jacquesm on Tuesday October 20, 2015 @01:44PM (#10420602)
Do you respect robots.txt? Do you publish your IP ranges?
Re:Show HN: Apifier – hosted web crawler for developers by jakubbalada on Tuesday October 20, 2015 @01:52PM (#10420642)
Yes, by default we respect robots.txt. There is a switch to disable it - on your own responsibility. We don't fully respect Crawl-delay, but minimum delay between requests for all our crawlers is set to 2000ms. We don't publish our IP ranges yet.