News for hackers. Stuff that matters.

A comment on Show HN: A Python Spider System with Web UI. Parent comment

Re:Show HN: A Python Spider System with Web UI by binux on Monday November 17, 2014 @08:27AM (#8618072)
http://demo.pyspider.org/debug/js_test_sciencedirect is a sample for this.

There is a phantomjs fetcher that can render the page as WebKit did. Furthermore, you can have some JavaScript running before/after page loaded to simulate a mouse click.

[ Parent | on HN ]
Re:Show HN: A Python Spider System with Web UI by pknerd on Monday November 17, 2014 @12:24PM (#8619213)
But will it not be slow? Assuming downloading css/images etc?
[ Parent | on HN ]
Re:Show HN: A Python Spider System with Web UI by binux on Monday November 17, 2014 @12:47PM (#8619339)
Images not downloaded default. Both the fetcher and the phantomjs proxy is totally async.
[ Parent | on HN ]