News for hackers. Stuff that matters.
Re:Portia, an open-source visual web scraper by egb on Tuesday April 01, 2014 @12:42PM (#7509598)
Anybody know of any tools that would work with JS-rendered sites, and not have to "parse the JS"?
Re:Portia, an open-source visual web scraper by egb on Tuesday April 01, 2014 @12:52PM (#7509693)
Answering my own question:

CasperJS is an open source navigation scripting & testing utility written in Javascript for the PhantomJS WebKit headless browser and SlimerJS (Gecko). It eases the process of defining a full navigation scenario and provides useful high-level functions, methods & syntactic sugar for doing common tasks such as:

    defining & ordering browsing navigation steps
    filling & submitting forms
    clicking & following links
    capturing screenshots of a page (or part of it)
    testing remote DOM
    logging events
    downloading resources, including binary ones
    writing functional test suites, saving results as JUnit XML
    scraping Web contents
Re:Portia, an open-source visual web scraper by mopoke on Wednesday April 02, 2014 @04:42AM (#7514282)
I wrote a blog post on my experiences using CasperJS to parse a single page site which used angular. http://www.andykelk.net/tech/web-scraping-with-casperjs
Re:Portia, an open-source visual web scraper by CMCDragonkai on Thursday April 03, 2014 @02:03AM (#7521221)
I recently created a service designed to make JS sites crawlable by search engines and other robots. However it works for scraping as well. Try the demo: https://snapsearch.io/
Re:Portia, an open-source visual web scraper by checker659 on Tuesday April 01, 2014 @12:45PM (#7509626)
PhantomJS?
Re:Portia, an open-source visual web scraper by e1g on Tuesday April 01, 2014 @12:54PM (#7509704)
>Is it also a webscraper that can pull data out of a page for me?

No, Phantom will only recreate the page as it would look like to a human user (i.e. after all javascript is parsed and executed). It will not help you parse or slice the page - you would have to do that part programatically with other dom-parsing tools.

Re:Portia, an open-source visual web scraper by checker659 on Tuesday April 01, 2014 @01:25PM (#7509995)
Sorry. I guess you're right. As a programmer, both of those look the same to me.
Re:Portia, an open-source visual web scraper by egb on Tuesday April 01, 2014 @12:48PM (#7509659)
"PhantomJS is a headless WebKit scriptable with a JavaScript API", so it's a browser.

Is it also a webscraper that can pull data out of a page for me?

Re:Portia, an open-source visual web scraper by nols on Tuesday April 01, 2014 @12:54PM (#7509706)
I've heard of people using PhantomJS with CasperJS to scrape, not sure if it can be done solely with PhantomJS.
Re:Portia, an open-source visual web scraper by vaviloff on Tuesday April 01, 2014 @02:21PM (#7510494)
CasperJS is a higher-level wrapper for PhantomJS, so - yes, it could be done with PhantomJS solely... But you wouldn't want to, because CasperJS makes automation easier.
Re:Portia, an open-source visual web scraper by uptown on Tuesday April 01, 2014 @02:37PM (#7510633)
Are there any libraries to facilitate database-connectivity to SQL Server or MySQL from javascript? I've used CasperJS to scrape some sites, but always fall back on post-processing the scraped data with another program in order to get it into my database. I'd love to be able to do it all from one piece of code.
Re:Portia, an open-source visual web scraper by gee_totes on Tuesday April 01, 2014 @04:56PM (#7511610)
You could always have your CasperJS scraping script make an AJAX request to a RESTful API for your MySQL DB. You won't be doing it all from one piece of code, but you'll be doing about 90% of it.
Re:Portia, an open-source visual web scraper by lost_my_pwd on Tuesday April 01, 2014 @12:49PM (#7509664)
SlimerJS too