News for hackers. Stuff that matters.
Re:Web Scraping 101 with Python by ankimal on Sunday March 10, 2013 @06:46PM (#5353877)
Also recommend Mechanize for Ruby (uses nokogiri under the covers).

http://mechanize.rubyforge.org/

http://phantomjs.org/ or "its easier to deal with cousin" http://casperjs.org/ for very client heavy sites.

FWIW, you can get away with HTML only scrapers most of the time, you just need to look harder to find all the data. Totally recommend using "View page source" as that would always give you the original HTML vs the possibly altered DOM (after JS has run on the page) that you might see with Dev Tools/Firebug.

Re:Web Scraping 101 with Python by johnnyg on Sunday March 10, 2013 @07:05PM (#5353967)
Mechanize is far and away the best and easiest way to scrape with Ruby until anything is rendered in javascript, which is explicitly not supported.

I tend to use Mechanized until I can't, then switch to Watir. Over time, I've found myself just strait up picking up Watir as it runs your browser directly and supports javascript rendering as a result.

Re:Web Scraping 101 with Python by ankimal on Sunday March 10, 2013 @08:21PM (#5354231)
How is performance with Watir? With casperjs a page takes me on an avg. 5-10 secs. to process.
Re:Web Scraping 101 with Python by johnnyg on Friday March 22, 2013 @01:17AM (#5421492)
Not great. About the same...
Re:Web Scraping 101 with Python by rscale on Sunday March 10, 2013 @07:42PM (#5354106)
I recommend Selenium before I'd recommend PhantomJS in situations where Mechanize/Nokogiri don't cut the mustard,

I've found Selenium scripts much easier to comprehend, modify, and maintain over time than the PhantomJS scripts.

Re:Web Scraping 101 with Python by ankimal on Sunday March 10, 2013 @08:22PM (#5354232)
Check out casperjs, it should make life easier. Phantomjs by itself is extremely cumbersome in my experience.