Re:Mechanize 0.2.0 released
by olaf on Saturday April 24, 2010 @10:42AM
(#1290916)
Unfortunately, Perl-WWW::Mechanize does not yet handle Javascript, has anybody (Perl, Python, Ruby) a solution for that? DOM+Javascript simulation comes to my mind. Or maybe a fully controllable browser component (where DOM and JS are built-in) with an API. Anyone with first-hand experience?
Re:Mechanize 0.2.0 released
by hugs on Saturday April 24, 2010 @01:20PM
(#1291186)
Things like Mechanize and HTMLUnit are what I call "HTTP-only protocol drivers" and are not a true-fidelity simulation of the HTML+JavaScript environment. This is exactly the reason I created Selenium in the first place. I had regression bugs in IE and Mozilla that I needed to verify were fixed in the actual browsers. An emulated browser environment like Mechanize or HTMLUnit isn't a real browser that users actually use, so it doesn't solve the regression testing problem I had.
Best of both worlds these days is in driving a headless WebKit (a true browser driver that includes the JS interaction), which will be coming in Selenium 2.
Re:Mechanize 0.2.0 released
by olaf on Saturday April 24, 2010 @03:41PM
(#1291458)
Thanks, Selenium can replay user interaction sessions, but I need to set e.g. form fields programmatically, is that possible? Maybe CPAN/Gtk2::WebKit is also an option.
Re:Mechanize 0.2.0 released
by tomjen3 on Saturday April 24, 2010 @06:08PM
(#1291653)
Internet Explorer.
No seriously, IE is fully controllable via COM and you can use pretty much any scripting language you want (the python COM api is pretty nice, so is the ruby one).
Re:Mechanize 0.2.0 released
by mumrah on Saturday April 24, 2010 @10:42AM
(#1290915)
I imagine you could use python-spidermonkey to execute isolated bits of JS. But yes, getting the window object into context in python would be super useful
Re:Mechanize 0.2.0 released
by alttab on Saturday April 24, 2010 @10:12AM
(#1290865)
We use the mechanize ruby gem for integration with other sites and web applications - doing things that are not exposed through APIs but only interface. Really cool!
To get a quick glimpse of how easy it is to work with look at this screencast: http://railscasts.com/episodes/191-mechanize
Best of both worlds these days is in driving a headless WebKit (a true browser driver that includes the JS interaction), which will be coming in Selenium 2.
http://www.layeredthoughts.com/automation/how-to-write-your-...
No seriously, IE is fully controllable via COM and you can use pretty much any scripting language you want (the python COM api is pretty nice, so is the ruby one).
Steve Yeggey had a post on the subject some time ago: http://sites.google.com/site/steveyegge2/scripting-windows-a...
BTW, MS Office has a similar feature.
I went the route: Mechanize -> HtmlUnit (from inside Clojure to be more scriptable) -> ended with Celerity and most happy.
It's a Python scraping framework based on Twisted. It lets you use some of Mechanize's API to describe the scraping jobs.
Now, if it only had a way to execute javascript....
Way better than Net::HTTP