Every story and comment from the Hacker News API, in a layout borrowed from 2001.
A comment on Show HN: Apifier – hosted web crawler for developers. Parent comment
- implement a mechanism to find and click active page elements, track the browser actions
- recompile PhantomJS to support POST requests
- implement a page queue with checks for duplicate URLs
- implement some parallelization and failover mechanism for case PhantomJS crashes (it does)
- possibly implement support for infinite scroll
- setup your own pool of proxy servers
- setup a database to store the results
and finally make the whole thing simple to setup and use :)
There are no comments on this comment yet.
- implement a mechanism to find and click active page elements, track the browser actions
- recompile PhantomJS to support POST requests
- implement a page queue with checks for duplicate URLs
- implement some parallelization and failover mechanism for case PhantomJS crashes (it does)
- possibly implement support for infinite scroll
- setup your own pool of proxy servers
- setup a database to store the results
and finally make the whole thing simple to setup and use :)