RoboBrowser: Your friendly neighborhood web scraper

Re:RoboBrowser: Your friendly neighborhood web scraper by aexaey on Thursday May 12, 2016 @04:48AM (#11682162)

I'm surprised nobody has mentioned WWW::Mechanize - classic perl library [1] or python port of it [2], which is much closer to RoboBrowser than selenium/phantomjs/horseman.

[1] http://search.cpan.org/~ether/WWW-Mechanize-1.75/lib/WWW/Mec...

[2] https://pypi.python.org/pypi/mechanize/

[ Parent | on HN ]

Re:RoboBrowser: Your friendly neighborhood web scraper by b3b0p on Thursday May 12, 2016 @10:35PM (#11688259)

I've used the Perl version a bit and loved it many years ago, but I've also had great success more recently with the Ruby [0] version of Mechanize.

[0] https://github.com/sparklemotion/mechanize

[ Parent | on HN ]

Re:RoboBrowser: Your friendly neighborhood web scraper by Dolores12 on Thursday May 12, 2016 @05:36AM (#11682311)

Mechanize is outdated and python 2 only. We tried it and switched to RoboBrowser.

[ Parent | on HN ]

Re:RoboBrowser: Your friendly neighborhood web scraper by grewil on Thursday May 12, 2016 @02:12PM (#11685517)

I find WWW::Mechanize::Firefox an interesting alternative. The API is similar to WWW::Mechanize, but it remote controls an instance of Firefox using the MozRepl addon. This means scraping/interacting with pages using JS works fine.

http://search.cpan.org/~corion/WWW-Mechanize-Firefox-0.78/li...

[ Parent | on HN ]

Re:RoboBrowser: Your friendly neighborhood web scraper by jimmaswell on Thursday May 12, 2016 @02:26PM (#11685617)

In some significant ways, python 2 is the better language than 3, though.

[ Parent | on HN ]

Re:RoboBrowser: Your friendly neighborhood web scraper by tmerr on Thursday May 12, 2016 @02:32PM (#11685663)

In what ways?

[ Parent | on HN ]

Re:RoboBrowser: Your friendly neighborhood web scraper by popey456963 on Thursday May 12, 2016 @02:59PM (#11685861)

Python 3 fan myself, but there are two things I've heard from colleagues:

1. More modules for Python2 than Python3. A lot of projects are forced to go down to Python2 to allow them to use the modules they want to use. 2. It's what they're familiar with. Generally the older developers like using Python2 because they know all what they're getting, no matter what strings are attached. 3. Better syntax. Apparently some people enjoy the Python2 syntax more than the Python3 one. Not using brackets for print statements seem to be the biggest plus, even though in my opinion it looks more Pythonic.

[ Parent | on HN ]