News for hackers. Stuff that matters.
Re:RoboBrowser: Your friendly neighborhood web scraper by markbnj on Thursday May 12, 2016 @09:28AM (#11683196)
Does this run javascript on the page? I've done quite a bit of scraping with scrapy, and have had to use phantomjs in many cases because static html doesn't get what you're after.
Re:RoboBrowser: Your friendly neighborhood web scraper by tekacs on Thursday May 12, 2016 @09:40AM (#11683266)
At a glance, no - it uses Requests to fetch pages and BeautifulSoup to parse them, the latter of which only parses the HTML into a document object.

So static HTML parsing only.

Re:RoboBrowser: Your friendly neighborhood web scraper by yes_or_gnome on Thursday May 12, 2016 @11:23AM (#11684049)
I use PhantomJS as well, but (assuming you haven't already) look at CasperJS. It uses PhantomJS, but it is more friendly to use for bigger tasks.
Re:RoboBrowser: Your friendly neighborhood web scraper by ddebernardy on Thursday May 12, 2016 @02:27PM (#11685633)
You could use Splash for JS as well. (Disclaimer: working for Scrapinghub, the main maintainers of Scrapy/Splash.)