News for hackers. Stuff that matters.

A comment on Python Headless Web Browser Scraping on Amazon Linux.

Re:Python Headless Web Browser Scraping on Amazon Linux by diminoten on Monday June 17, 2013 @12:25PM (#5894060)
I find it preferable to determine the requests that jQuery is making and perform them myself to extract the necessary data, rather than load up a whole browser just to do the same thing.

Selenium is terrible, performance wise, and requires a significant investment in environment in order to work reliably. I try to avoid it except when I absolutely cannot.

[ Parent | on HN ]
Re:Python Headless Web Browser Scraping on Amazon Linux by ArbitraryCrow on Tuesday June 18, 2013 @10:03AM (#5899182)
I wound up doing this myself, after spending an undue amount of time struggling with a morass of insanely written Javascript. Fiddler proved indispensable for observing the actual interaction with the web server.
[ Parent | on HN ]