Re:I Don’t Need No Stinking API: Web Scraping For Fun and Pr...
by frabcus on Sunday December 09, 2012 @07:04AM
(#4894531)
I've started playing with zombie.js recently as well - much lighter and faster than the ones that instrument a completely full browser. But has a full Javascript engine.
Re:I Don’t Need No Stinking API: Web Scraping For Fun and Pr...
by freshhawk on Sunday December 09, 2012 @03:06PM
(#4895906)
That's really interesting, thanks.
I worry that it's not going to replicate a real browser accurately enough, but I'm excited to try it out a bit.
Re:I Don’t Need No Stinking API: Web Scraping For Fun and Pr...
by jonpaul on Sunday December 09, 2012 @03:46PM
(#4896060)
Your worry is correct. http://news.ycombinator.com/item?id=4896054 I've tried scraping with it, and it failed miserably on some sites.
Re:I Don’t Need No Stinking API: Web Scraping For Fun and Pr...
by frabcus on Monday December 10, 2012 @07:59AM
(#4898782)
Yeah, it's not mature enough yet.
We're also trying it for integration tests, as it is much quicker than Phantom or Selenium. Even there, where we control the standards-compliant site, it isn't quite good enough yet.
Would love to see more people helping make it so, though!
Re:I Don’t Need No Stinking API: Web Scraping For Fun and Pr...
by jonpaul on Sunday December 09, 2012 @03:45PM
(#4896054)
zombie.js is not a full browser. It's a poor emulation using jsdom as its backing. http://zombie.labnotes.org/guts Beware, for some applications, jsdom is super buggy.
If you can't get the data from the endpoints the javascript hits then write your scraper in javascript and have it run in a headless browser, and it's the webkit engine so most sites test their site against it heavily.
Either pull the data out of the javascript objects or trigger your extraction from the html by attaching to the events in the javascript.