News for hackers. Stuff that matters.
Re:Web Scraping 101 with Python by thrownaway2424 on Sunday March 10, 2013 @05:06PM (#5353463)
But why python? Stop now and use perl. Perl's HTTP libraries are actually sane, unlike urllib[2], WWW::Mechanize is brilliant, and it's easier to throw in disgusting hacks in perl when you need them, which is constantly in the business of scraping the web.
Re:Web Scraping 101 with Python by cwgem on Sunday March 10, 2013 @06:15PM (#5353752)
Because the writer is accustomed to Python as their language of choice? Maybe it's a Python shop? What if they want to integrate it into their Django site?

If you're coming from a primarily Python background you won't just be waltzing right in and using CPAN modules right away. You have to understand the underlying language as well. Python and perl's object oriented systems for example are quite different (using Moose helps somewhat granted the person knows it even exists). Then there's the issue of understanding various contexts. These differences may take time getting used to depending on the level of the developer.

Python also supports regex as well, which can be useful for weird situations. Granted however it's not going to be as tightly integrated as perl. There's even a Natural Language Toolkit if you want to get really crazy with things.

TLDR: Right tool for the job should take environmental circumstances into account as well

Re:Web Scraping 101 with Python by slurgfest on Sunday March 10, 2013 @06:34PM (#5353841)
Perl is valid if you know it and want to use it. Otherwise, not really.

There is a port of mechanize to Python as well, if that is your main reason to use Perl.

Whatever you can do in "disgusting hacks" can be done just as quickly in a way which won't make you want to vomit when you look over the code again later.

Re:Web Scraping 101 with Python by creaktive on Sunday March 10, 2013 @06:35PM (#5353846)
I bet Perl-based scrapers run faster ;) http://news.ycombinator.com/item?id=5252581
Re:Web Scraping 101 with Python by niels on Sunday March 10, 2013 @05:31PM (#5353564)
There are other python libraries than urllib.
Re:Web Scraping 101 with Python by the_cat_kittles on Sunday March 10, 2013 @06:24PM (#5353800)
requests is an excellent library to use in place of urllib