Re:RoboBrowser: Your friendly neighborhood web scraper
by markbnj on Thursday May 12, 2016 @09:28AM
(#11683196)
Does this run javascript on the page? I've done quite a bit of scraping with scrapy, and have had to use phantomjs in many cases because static html doesn't get what you're after.
[ Parent
| on HN ]
Re:RoboBrowser: Your friendly neighborhood web scraper
by tekacs on Thursday May 12, 2016 @09:40AM
(#11683266)
At a glance, no - it uses Requests to fetch pages and BeautifulSoup to parse them, the latter of which only parses the HTML into a document object.
So static HTML parsing only.
Re:RoboBrowser: Your friendly neighborhood web scraper
by yes_or_gnome on Thursday May 12, 2016 @11:23AM
(#11684049)
I use PhantomJS as well, but (assuming you haven't already) look at CasperJS. It uses PhantomJS, but it is more friendly to use for bigger tasks.
Re:RoboBrowser: Your friendly neighborhood web scraper
by ddebernardy on Thursday May 12, 2016 @02:27PM
(#11685633)
You could use Splash for JS as well. (Disclaimer: working for Scrapinghub, the main maintainers of Scrapy/Splash.)