Re:Web Scraping with Beautiful Soup (2014)
by Animats on Friday November 11, 2016 @02:16PM
(#12933438)
Cute. That's what BeautifulSoup is good for.
I'm a longtime user of BeautifulSoup.
BeautifulSoup does not use or create "the DOM". It does convert HTML into a tree, but that tree is somewhat different from a browser's Document Object Model. For most screen-scraping purposes, this doesn't matter. But if the page uses Javascript to manipulate the DOM on page load, it does.
I have a tool for looking at a web page through BeautifulSoup. This reads the page from a server, parses it into a tree with BeautifulSoup using the HTML5 parser, discards all Javascript, makes all links absolute, and turns the tree back into HTML in UTF-8, properly indented. If you run a page through this and it still makes sense, scraping will probably work. If not, simple scraping won't work and you'll probably have to use a program-controlled browser that will execute JavaScript.
I'm a longtime user of BeautifulSoup.
BeautifulSoup does not use or create "the DOM". It does convert HTML into a tree, but that tree is somewhat different from a browser's Document Object Model. For most screen-scraping purposes, this doesn't matter. But if the page uses Javascript to manipulate the DOM on page load, it does.
I have a tool for looking at a web page through BeautifulSoup. This reads the page from a server, parses it into a tree with BeautifulSoup using the HTML5 parser, discards all Javascript, makes all links absolute, and turns the tree back into HTML in UTF-8, properly indented. If you run a page through this and it still makes sense, scraping will probably work. If not, simple scraping won't work and you'll probably have to use a program-controlled browser that will execute JavaScript.
Some examples:
HN looks good: http://www.sitetruth.com/fcgi/viewer.fcgi?url=https://news.y...
AFL-CIO, the site used in the article, looks great: http://www.sitetruth.com/fcgi/viewer.fcgi?url=aflcio.org
Twitter's images disappear: http://www.sitetruth.com/fcgi/viewer.fcgi?url=https://www.tw...
Adobe's formatting disappears: http://www.sitetruth.com/fcgi/viewer.fcgi?url=https://www.ad...
Intel complains about the browser but looks OK: http://www.sitetruth.com/fcgi/viewer.fcgi?url=intel.com
Grubhub gives us nothing as plain HTML: http://www.sitetruth.com/fcgi/viewer.fcgi?url=grubhub.com
Same for Doordash: http://www.sitetruth.com/fcgi/viewer.fcgi?url=doordash.com
(No scraping restaurant menus with BeautifulSoup.)
Cool stuff in pure CSS works fine: http://www.sitetruth.com/fcgi/viewer.fcgi?url=css3.bradshawe...
(You don't really need Javascript any more just to get the page up.)