Re:Web crawling and downloading ebooks with phantomJS
by rb2k_ on Sunday June 17, 2012 @11:46AM
(#4123143)
When you say 'rely on jQuery', I think it would be more precise to say that it relies on CSS selectors. Most libraries will actually provide you a way to access an HTML parse tree in that way.
I guess phantomjs is a good a tool as any, but there is really no need to evaluate Javascript for a bit of plain HTTP+HTML parsing.
Re:Web crawling and downloading ebooks with phantomJS
by mistercow on Sunday June 17, 2012 @01:41PM
(#4123376)
It looks to me like it's using jQuery specific functions. It could be done with a simpler selector engine, but in this case, it looks pretty clear that it's either jQuery or a compatible library like Zepto.
Re:Web crawling and downloading ebooks with phantomJS
by rb2k_ on Sunday June 17, 2012 @08:43PM
(#4124350)
As far as I see the only line with selectors is:
> return [ $($('h2 a')[0]).attr('title'), $($('h2 a')[1]).attr('title') ];
which are 2 css selectors and picking an element. That is pretty much covered by all of the available http parser libraries
Re:Web crawling and downloading ebooks with phantomJS
by mistercow on Sunday June 17, 2012 @11:51PM
(#4124880)
It's CSS selectors and then wrapping the DOM element again in a function to give it an `attr` method, which is jQuery style. Other libraries may use that syntax too, but I'm pretty sure it started with jQuery (and if not, was certainly popularized by it).
One advantage is that it's not always instantly obvious if you'll need JS to execute before you can scrape a page. If you start out with a simply html parser and then find out that you needed the JS to run first, you're going to have to start over. If you start out using phantomJS and then find out that you don't need any of the original JS to run, your script still works.