Scrappy - Simple Perl Scraping Framework

Re:Scrappy - Simple Perl Scraping Framework by arkitaip on Sunday March 06, 2011 @12:11PM (#2294494)

I was looking at several scraping solutions (e.g. imacros, selenium) that can handle DHTML for a project and they all have significant performance issues since they need to render the actual pages before processing them. A couple of thousands or rows isn't a problem but try anything more and you got a real performance bottleneck.

[ Parent | on HN ]

Re:Scrappy - Simple Perl Scraping Framework by odonnell on Sunday March 06, 2011 @05:14PM (#2295181)

DHTML is server-side. You mean AJAX. Also, think of the page as an interface to a more lightweight web service. You should probably be parsing that directly.

[ Parent | on HN ]

Re:Scrappy - Simple Perl Scraping Framework by thirdusername on Monday March 07, 2011 @09:56AM (#2297269)

He's referring to this: http://en.wikipedia.org/wiki/Dhtml I'm not sure what DHTML you are thinking of that would be server-side.

[ Parent | on HN ]

Re:Scrappy - Simple Perl Scraping Framework by odonnell on Wednesday March 09, 2011 @01:15AM (#2304079)

Fuck, thinking of SHTML for some reason.

[ Parent | on HN ]

Re:Scrappy - Simple Perl Scraping Framework by chsonnu on Sunday March 06, 2011 @09:10PM (#2295698)

Have you tried Watir? I'm not sure if it'll solve your performance issue, but it's been at least twice as fast as Selenium for me.

[ Parent | on HN ]