Re:Better Python Scraping - Installing lxml and Beautiful Soup
by wesleyzhao on Tuesday June 28, 2011 @09:35AM
(#2705308)
Interesting - will have to play with it when I wake up. Any reason in particular you think it is better than lxml? Beautiful Soup is only used with lxml to parse broken HTML into a DOM. Lxml does most of the heavy lfiting
[ Parent
| on HN ]
Re:Better Python Scraping - Installing lxml and Beautiful Soup
by cdr on Tuesday June 28, 2011 @09:46AM
(#2705355)
Scrapy is a django-like framework. It has more structure and a lot of built-in features compared to using a library like lxml.
lxml and Scrapy use the same library on the backend, libxml2.
http://doc.scrapy.org/faq.html#how-does-scrapy-compare-to-be... http://doc.scrapy.org/topics/selectors.html
Re:Better Python Scraping - Installing lxml and Beautiful Soup
by wesleyzhao on Tuesday June 28, 2011 @09:57AM
(#2705386)
If its a framework does it take a little longer to setup/get used to? If so is the learning curve worth it?
Re:Better Python Scraping - Installing lxml and Beautiful Soup
by cdr on Tuesday June 28, 2011 @10:04AM
(#2705411)
I can't make a fair comparison as far as learning curve, but if you're new to scraping Scrapy might in theory be easier. You get a bunch of tools built-in that you don't have to come up with yourself, and you get a pre-defined way of doing things. However, last time I checked the example project in the docs was a little lacking and there's not many good external examples either.
I personally really like having the structure of a framework. It lets me churn out simple project from boilerplate very quickly, and it helps keep larger projects organized.