News for hackers. Stuff that matters.
Re:I Don’t Need No Stinking API: Web Scraping For Fun and Pr... by bdcravens on Sunday December 09, 2012 @01:14AM (#4894007)
Another issue not covered: file downloads. Let's say you have a process that creates a dynamic image, or logs in and downloads dynamic PDFs. Even Selenium can't handle this (the download dialog is an OS-level feature). At one point I was able to get Chrome to auto-download in Selenium, but had zero control over filename and where it was saving. I ended up using iMacros (the pay version) to drive this (using Windows instances: their Linux version is very immature comparably).
Re:I Don’t Need No Stinking API: Web Scraping For Fun and Pr... by dphase on Sunday December 09, 2012 @01:32AM (#4894043)
I've done this successfully with Ruby Mechanize.
Re:I Don’t Need No Stinking API: Web Scraping For Fun and Pr... by bdcravens on Sunday December 09, 2012 @01:56AM (#4894092)
Awesome. I'd love some hints or links, as I'm always looking to refactor.
Re:I Don’t Need No Stinking API: Web Scraping For Fun and Pr... by goostavos on Sunday December 09, 2012 @02:23AM (#4894141)
In general, if you're going the mechanize route, .retrieve() is the function your looking for.

e.g.

  br = mechanize.Browser()
  br.retrieve("https://www.google.com/images/srpr/logo3w.png, google_logo.png)[0]
Mechanize doesn't really have a proper doc, but just about everything you'd need could be figured out from the very lengthy examples page on their site.
Re:I Don’t Need No Stinking API: Web Scraping For Fun and Pr... by bdcravens on Sunday December 09, 2012 @02:38AM (#4894169)
Playing with it now, and while it seems to hit my download need, I can't seem to get it to play nice with sites that are JavaScript dependent. Am I missing something, or is there a way to plugin an underlying WebKit engine?
Re:I Don’t Need No Stinking API: Web Scraping For Fun and Pr... by bryogenic on Sunday December 09, 2012 @08:33AM (#4894692)
PhantomJS is capable of downloading binary content from js dependent sites but it is a journey to get it working as it is not an out-of-the-box feature. Instead use CasperJS to drive Phantom and get a ton of snazzy features including simple binary downloads. Happy scraping!