Show HN: A Python Spider System with Web UI

Re:Show HN: A Python Spider System with Web UI by maratc on Monday November 17, 2014 @11:05AM (#8618763)

Will it be a good fit if I, running on a hundred servers, need to scrape just the home page of a million sites? No analysis of the pages, that is done later.

[ Parent | on HN ]

Re:Show HN: A Python Spider System with Web UI by binux on Monday November 17, 2014 @11:23AM (#8618847)

The fetcher fit you already...

[ Parent | on HN ]

Re:Show HN: A Python Spider System with Web UI by maratc on Monday November 17, 2014 @11:42AM (#8618965)

You are running

   phantomjs phantomjs_fetcher.js

and using it as proxy? The setup instructions are a bit unclear on this.

[ Parent | on HN ]

Re:Show HN: A Python Spider System with Web UI by binux on Monday November 17, 2014 @11:52AM (#8619024)

I want to make it a http proxy in the beginning. But I found it hard to do so. Then I post every to it, but haven't change the name.

But it works like a proxy, that any request with `fetch_type == 'js'` would be fetched through phantomjs and the response back to tornado_fetcher.

[ Parent | on HN ]