Re:Web Scraping 101 with Python
by pmorici on Sunday March 10, 2013 @05:51PM
(#5353648)
BeautifulSoup, has received a lot of positive press on HN over the years so when I needed to do some heavy scraping I gave it a spin. It was a total disappointment. It's fine if you are scraping a small set of similar pages from a single site but if you are scraping a large number of pages across many sites and esp pages with text encoding other than ascii / utf-8 it choaks so frequently as to be useless. BeautifulSoup is fine for small jobs but if you were making a web-crawler for example look elsewhere it is totally inadequate.
[ Parent
| on HN ]
lxml.HTML (+ html5lib if needed) is a FAR superior choice indeed. It's an order of magnitude faster, and you can use not only xpath selectors, but also CSS selectors too, from a lxml dom.
or indeed you can use Scrapy, which fixes all these encoding BS, handles per domain crawl rate and concurrent requests etc...
> with text encoding other than ascii / utf-8 it choaks so frequently as to be useless
I had the opposite experience though I used it only "for small jobs".
If there are no Content-Type (either in http headers or meta http-equiv) that specifies character encoding, no meta charset, and no xml declaration for xhtml, etc that is if the only way to find out character encoding is to guess then even in this case BeautifulSoup includes UnicodeDammit that uses chardet to guess the encoding.
In my experience lxml.html is much better.
The article shows that BeautifulSoup can use lxml internally. It can also use html5lib.
I'll second, lxml.html is in my experience very robust and fast. I've been writing a lot of scrapers along the years and the best combination I found so far is requests / lxml.html / gevent.
It doesn't get any simpler than this IMO http://pastebin.com/hacxmAjV
For fun here's the Ruby+Nokogiri version and my attempt at the Clojure+Enlive version (3rd day learning Clojure).
Pretty nice. I'd like to see how it goes in a real world scenario with concurrency, manipulation of extracted nodes and general HTTP post/auth/etc. I'm not sure about Ruby+Nokogiri but Clojure+Enlive may well be a great choice.
I don't see a use for BeautifulSoup nowadays. I use lxml.etree for everything (with the HTML parser when needed), and do 99% of queries using XPath. It's the best way to do scraping with Python, in my experience.
Re:Web Scraping 101 with Python
by morganwatch on Sunday March 10, 2013 @06:30PM
(#5353830)
what would you suggest?
Why not both?
http://www.crummy.com/software/BeautifulSoup/bs4/doc/#specif...
waiter there's some lxml in my soup!