News for hackers. Stuff that matters.

A comment on Show HN: Apifier – hosted web crawler for developers.

Re:Show HN: Apifier – hosted web crawler for developers by Eridrus on Tuesday October 20, 2015 @04:24PM (#10421588)
Why should I use this instead of just firing up some spot instances with phantomjs?
[ Parent | on HN ]
Re:Show HN: Apifier – hosted web crawler for developers by jancurn on Tuesday October 20, 2015 @04:45PM (#10421695)
You could definitely do that, but then you might also need to:

- implement a mechanism to find and click active page elements, track the browser actions

- recompile PhantomJS to support POST requests

- implement a page queue with checks for duplicate URLs

- implement some parallelization and failover mechanism for case PhantomJS crashes (it does)

- possibly implement support for infinite scroll

- setup your own pool of proxy servers

- setup a database to store the results

and finally make the whole thing simple to setup and use :)

[ Parent | on HN ]