News for hackers. Stuff that matters.

A comment on Web crawler.

Re:Web crawler by jafingi on Wednesday April 30, 2014 @02:49AM (#7671941)
You should read about screen scraping :-) That's the essence of web crawling.
[ Parent | on HN ]
Re:Web crawler by kachen on Friday May 02, 2014 @11:25PM (#7689434)
thank u bro.. Mr jafingi do u have any good refrence of screen scraping
[ Parent | on HN ]
Re:Web crawler by jafingi on Saturday May 03, 2014 @04:35AM (#7690075)
Well, you'd have to understand the fundamentals of DOM: http://www.w3.org/TR/DOM-Level-2-Core/introduction.html

Also learn about XPath, which is often used as a language to selecting nodes in documents.

Scrapy and BeautifulSoup are popular libraries for scraping in Python.

[ Parent | on HN ]