Ask HN: Crawler for my app. Please suggest

Posted by rajlalwani on Tuesday December 16, 2014 @11:18AM
from the ask-hn dept.

Hi, I am planning to write an app what requires web pages to be crawled. Could you pls suggest one? Thanks in advance.

( 3 comments | on HN )

Re:Ask HN: Crawler for my app. Please suggest by mtmail on Tuesday December 16, 2014 @11:32AM (#8757818)

Have a look at import.io as long as you're dealing with a small set of pages, e.g. ecommerce.

(I note you asked about git 2 weeks, then which programming language to use, then native vs html app, then which test framework to use, now which crawler to use. It makes it sound like you didn't do research on your own)

[ Parent | on HN ]

Re:Ask HN: Crawler for my app. Please suggest by wlrd on Tuesday December 16, 2014 @11:20AM (#8757757)

You could try out http://scrapy.org/.

[ Parent | on HN ]

Re:Ask HN: Crawler for my app. Please suggest by Jeremy1026 on Tuesday December 16, 2014 @11:20AM (#8757761)

Not sure what you are asking for, a crawler, or a website to be crawled.

[ Parent | on HN ]