Re:An Introduction to Compassionate Screen Scraping
by jerf on Tuesday April 19, 2011 @04:43PM
(#2464131)
It may not use the MIGHTY POWER OF JAVASCRIPT, but BeautifulSoup is a best-of-breed real-world HTML parser. Not just in the API, but in the verification that its parsing algorithm is effective against HTML found on real web sites. I find it unlikely that jsdom is actually significantly better. Does that code really look like it's going to be significantly improved with jQuery?
packs a lot of punch.
[ Parent
| on HN ]
Re:An Introduction to Compassionate Screen Scraping
by meatmanek on Tuesday April 19, 2011 @05:05PM
(#2464226)
The ability to use CSS/jQuery selectors is really nice, though; In order to find all <td>s whose parents have a class "blah", you have to use list comprehension:
tds = [td for td in soup.findAll('td') if td.parent.get('class') == 'blah']
In jQuery, this is more compactly written: tds = $('.blah > td')
And if you just want to look for <td>s somewhere within a .blah element, you can use tds = $('.blah td')
This is a lot less clear in BeautifulSoup: tds = [td for td in soup.findAll('td') if td.findParents(attrs={'class': 'blah'})]
(If there are better ways to write this BeautifulSoup code, please let me know)Selectors have some other benefits too - you can just go to the CSS file and grab the selector that matches what you want, and you can be reasonably sure it'll work in most cases.
Re:An Introduction to Compassionate Screen Scraping
by joshu on Wednesday April 20, 2011 @03:13AM
(#2465614)
Lxml in python lets you use CSS selectors.
Re:An Introduction to Compassionate Screen Scraping
by cdr on Tuesday April 19, 2011 @09:17PM
(#2464860)
My impression was that BeautifulSoup is actually getting very long in the tooth - I didn't use it much, but I have used a lot of Scrapy, a beautiful framework that completely supplants BeautifulSoup for scraping.