News for hackers. Stuff that matters.

A comment on Scrappy - Simple Perl Scraping Framework.

Re:Scrappy - Simple Perl Scraping Framework by mwexler on Sunday March 06, 2011 @12:01PM (#2294479)
Most scrapers seem to break down on heavy dynamic/ajax pages. For example, anything made with GWT appears to provide little for the average scraper to grab (say, for automated daily tracking of android app downloads, for example). Short of reversing out the foreign pages api calls, has anyone encountered a solution to do more processing and then scrape the rendered page?

(well, and short of using Selenium to script a login, then scrape the rendered page via a controlled firefox... which works, but is clunky)

[ Parent | on HN ]
Re:Scrappy - Simple Perl Scraping Framework by arkitaip on Sunday March 06, 2011 @12:11PM (#2294494)
I was looking at several scraping solutions (e.g. imacros, selenium) that can handle DHTML for a project and they all have significant performance issues since they need to render the actual pages before processing them. A couple of thousands or rows isn't a problem but try anything more and you got a real performance bottleneck.
[ Parent | on HN ]
Re:Scrappy - Simple Perl Scraping Framework by odonnell on Sunday March 06, 2011 @05:14PM (#2295181)
DHTML is server-side. You mean AJAX. Also, think of the page as an interface to a more lightweight web service. You should probably be parsing that directly.
[ Parent | on HN ]
Re:Scrappy - Simple Perl Scraping Framework by thirdusername on Monday March 07, 2011 @09:56AM (#2297269)
He's referring to this: http://en.wikipedia.org/wiki/Dhtml I'm not sure what DHTML you are thinking of that would be server-side.
[ Parent | on HN ]
Re:Scrappy - Simple Perl Scraping Framework by odonnell on Wednesday March 09, 2011 @01:15AM (#2304079)
Fuck, thinking of SHTML for some reason.
[ Parent | on HN ]
Re:Scrappy - Simple Perl Scraping Framework by chsonnu on Sunday March 06, 2011 @09:10PM (#2295698)
Have you tried Watir? I'm not sure if it'll solve your performance issue, but it's been at least twice as fast as Selenium for me.
[ Parent | on HN ]
Re:Scrappy - Simple Perl Scraping Framework by odonnell on Sunday March 06, 2011 @05:11PM (#2295174)
Use Charles or another proxy and find what feed the page is loading, then parse that. At least two requests need to be made anyway.
[ Parent | on HN ]