News for hackers. Stuff that matters.
Re:Better Python Scraping - Installing lxml and Beautiful Soup by cdr on Tuesday June 28, 2011 @09:32AM (#2705298)
I much prefer Scrapy (http://scrapy.org/). BeautifulSoup is pretty outdated.
Re:Better Python Scraping - Installing lxml and Beautiful Soup by jdq on Tuesday June 28, 2011 @04:50PM (#2707266)
I concur, but technically Scrapy is an entire web scraping/crawling framework for writing crawlers, not just XML/HTML parsing like BeautifulSoup or lxml. You don't even have to use Scrapy's built in processor, you can use BeautifulSoup (or whatever else) if you want. What Scrapy gets you is all the logic for crawling of the web pages (requesting pages, reacting to html errors, etc.). You basically just tell it what urls to parse, what to parse from the pages, and what to do with the parsed data. It handles all the rest. I used it Scrapy just recently on an online movie site (shameless plug: www.qwink.com).
Re:Better Python Scraping - Installing lxml and Beautiful Soup by iam on Tuesday June 28, 2011 @11:32AM (#2705768)
XPath/CSS path selectors for scraping are definitely the "now" way to do it. I recommend Nokogiri (http://nokogiri.org/) for the Ruby users, it does pretty much the same thing in a nutshell.

What it really comes down it is 2x-3x smaller code, plus much faster to write it since you can just test your CSS selector in your web browser such as with FireBug before sticking it into your code.

Re:Better Python Scraping - Installing lxml and Beautiful Soup by wesleyzhao on Tuesday June 28, 2011 @03:04PM (#2706796)
If I'm not mistaken LXML uses XPath and CSS path selectors right?
Re:Better Python Scraping - Installing lxml and Beautiful Soup by wesleyzhao on Tuesday June 28, 2011 @09:35AM (#2705308)
Interesting - will have to play with it when I wake up. Any reason in particular you think it is better than lxml? Beautiful Soup is only used with lxml to parse broken HTML into a DOM. Lxml does most of the heavy lfiting
Re:Better Python Scraping - Installing lxml and Beautiful Soup by cdr on Tuesday June 28, 2011 @09:46AM (#2705355)
Scrapy is a django-like framework. It has more structure and a lot of built-in features compared to using a library like lxml.

lxml and Scrapy use the same library on the backend, libxml2.

http://doc.scrapy.org/faq.html#how-does-scrapy-compare-to-be... http://doc.scrapy.org/topics/selectors.html

Re:Better Python Scraping - Installing lxml and Beautiful Soup by wesleyzhao on Tuesday June 28, 2011 @09:57AM (#2705386)
If its a framework does it take a little longer to setup/get used to? If so is the learning curve worth it?
Re:Better Python Scraping - Installing lxml and Beautiful Soup by cdr on Tuesday June 28, 2011 @10:04AM (#2705411)
I can't make a fair comparison as far as learning curve, but if you're new to scraping Scrapy might in theory be easier. You get a bunch of tools built-in that you don't have to come up with yourself, and you get a pre-defined way of doing things. However, last time I checked the example project in the docs was a little lacking and there's not many good external examples either.

I personally really like having the structure of a framework. It lets me churn out simple project from boilerplate very quickly, and it helps keep larger projects organized.

Re:Better Python Scraping - Installing lxml and Beautiful Soup by wladimir on Tuesday June 28, 2011 @09:42AM (#2705337)
How is BeautifulSoup "outdated"? It's meant for making it easy to scrape non-properly-formatted sites, and works very well at that, as I've found out in practice.

What's the advantage of Scrapy?

Re:Better Python Scraping - Installing lxml and Beautiful Soup by cdr on Tuesday June 28, 2011 @09:50AM (#2705364)
http://www.crummy.com/software/BeautifulSoup/3.1-problems.ht...

I guess it looks like BeautifulSoup finally got a 4.0 alpha release out which supposedly works, but that took several years. The codebase is aged and releases are incredibly slow.

"I no longer enjoy working on Beautiful Soup..."

"Parsing is longer a competitive advantage for Beautiful Soup, and I'd be happier if I could get out of the parser business altogether."

Re:Better Python Scraping - Installing lxml and Beautiful Soup by wesleyzhao on Tuesday June 28, 2011 @09:58AM (#2705391)
Very interesting... I typically would think you should not use a product that a founder/developer does not believe in