Web crawler

Posted by kachen on Wednesday April 30, 2014 @01:41AM
from the self-post dept.

what is the best refrence for studying basic we crawling?

( 5 comments | 2 points | on HN )

Re:Web crawler by wlrd on Wednesday April 30, 2014 @01:52AM (#7671821)

If you have an urge to work in python, I would strongly recommend Scrapy. You could also use Beautiful Soup if you wanted. BS would be a little harder to use but is very useful.

[ Parent | on HN ]

Re:Web crawler by kachen on Friday May 02, 2014 @11:22PM (#7689425)

thank u bros, actually I am planning to make a livescore app. in android and need online feeds for it.. using API seems limited and I don't know any thing of web crawling...So suggest what should I do.

[ Parent | on HN ]

Re:Web crawler by jafingi on Wednesday April 30, 2014 @02:49AM (#7671941)

You should read about screen scraping :-) That's the essence of web crawling.

[ Parent | on HN ]

Re:Web crawler by kachen on Friday May 02, 2014 @11:25PM (#7689434)

thank u bro.. Mr jafingi do u have any good refrence of screen scraping

[ Parent | on HN ]

Re:Web crawler by jafingi on Saturday May 03, 2014 @04:35AM (#7690075)

Well, you'd have to understand the fundamentals of DOM: http://www.w3.org/TR/DOM-Level-2-Core/introduction.html

Also learn about XPath, which is often used as a language to selecting nodes in documents.

Scrapy and BeautifulSoup are popular libraries for scraping in Python.

[ Parent | on HN ]