Re:Web Scraping with Beautiful Soup (2014)
by prions on Friday November 11, 2016 @03:21PM
(#12933969)
I recently used BeautifulSoup in a Wikipedia scraping project. It's definitely a great tool but it had a few annoying functionality issues.
I had some preprocessing using decompose(), which can take lists of attributes. However it cannot decompose multiple attributes at once such as anchors with a certain title and table elements in one call. It felt cumbersome to call it multiple times.
It seems like Scrapy does exactly what I needed though (click on the first link in the article), which would have saved me a ton of headache. So it goes.
I had some preprocessing using decompose(), which can take lists of attributes. However it cannot decompose multiple attributes at once such as anchors with a certain title and table elements in one call. It felt cumbersome to call it multiple times.
It seems like Scrapy does exactly what I needed though (click on the first link in the article), which would have saved me a ton of headache. So it goes.