Re:Show HN: Apifier – hosted web crawler for developers
by misiti3780 on Tuesday October 20, 2015 @01:59PM
(#10420691)
I like the idea - would probably use it in the future - can you talk a little bit about what technologies you are using?
[ Parent
| on HN ]
Re:Show HN: Apifier – hosted web crawler for developers
by jancurn on Tuesday October 20, 2015 @02:07PM
(#10420745)
Thank you! The crawler is currently based on PhantomJS, there is a pool of worker nodes that distribute the crawling to more machines. We use Node.js + MongoDB backend and Meteor for front-end.