Show HN: A Python Spider System with Web UI

Re:Show HN: A Python Spider System with Web UI by _bitliner on Monday November 17, 2014 @08:16AM (#8618013)

Furthermore, what you mean with `Javascript pages supported`? Could I just specify where it has to click or do I need to make a reverse engineering of the ajax calls?

[ Parent | on HN ]

Re:Show HN: A Python Spider System with Web UI by binux on Monday November 17, 2014 @08:27AM (#8618072)

http://demo.pyspider.org/debug/js_test_sciencedirect is a sample for this.

There is a phantomjs fetcher that can render the page as WebKit did. Furthermore, you can have some JavaScript running before/after page loaded to simulate a mouse click.

[ Parent | on HN ]

Re:Show HN: A Python Spider System with Web UI by pknerd on Monday November 17, 2014 @12:24PM (#8619213)

But will it not be slow? Assuming downloading css/images etc?

[ Parent | on HN ]

Re:Show HN: A Python Spider System with Web UI by binux on Monday November 17, 2014 @12:47PM (#8619339)

Images not downloaded default. Both the fetcher and the phantomjs proxy is totally async.

[ Parent | on HN ]