Re:Better Python Scraping - Installing lxml and Beautiful Soup
by wesleyzhao on Tuesday June 28, 2011 @09:57AM
(#2705386)
If its a framework does it take a little longer to setup/get used to? If so is the learning curve worth it?
Re:Better Python Scraping - Installing lxml and Beautiful Soup
by cdr on Tuesday June 28, 2011 @10:04AM
(#2705411)
I can't make a fair comparison as far as learning curve, but if you're new to scraping Scrapy might in theory be easier. You get a bunch of tools built-in that you don't have to come up with yourself, and you get a pre-defined way of doing things. However, last time I checked the example project in the docs was a little lacking and there's not many good external examples either.
I personally really like having the structure of a framework. It lets me churn out simple project from boilerplate very quickly, and it helps keep larger projects organized.
lxml and Scrapy use the same library on the backend, libxml2.
http://doc.scrapy.org/faq.html#how-does-scrapy-compare-to-be... http://doc.scrapy.org/topics/selectors.html