News for hackers. Stuff that matters.
Re:Better Python Scraping - Installing lxml and Beautiful Soup by tsumnia on Tuesday June 28, 2011 @11:49AM (#2705841)
Is mechanize (http://wwwsearch.sourceforge.net/mechanize/) considered outdated or convoluted? It's what I've used for my scrapings.

Also, how well do these other scrapers handle Javascript? I've had to abandon some scrapes from ASP pages because they wouldn't properly handle it.

Re:Better Python Scraping - Installing lxml and Beautiful Soup by danohuiginn on Tuesday June 28, 2011 @01:45PM (#2706350)
it's a different tool. Mechanize concentrates on navigating pages and downloading pages, following links, handling cookies etc. BeautifulSoup and lxml parse information out of the html.

There's some overlap, but not much. I have tended to use BeautifulSoup and mechanize together. As mentioned above, BeautifulSoup is no longer being actively maintained, and I'd recommend starting with lxml in most cases. I'm still using BeautifulSoup mainly because I have most of the package memorized.

Re:Better Python Scraping - Installing lxml and Beautiful Soup by tsumnia on Tuesday June 28, 2011 @02:05PM (#2706453)
Thanks, I use the same combination, but haven't needed to make any new parses in a while. I'll read up on lxml for the next time.
Re:Better Python Scraping - Installing lxml and Beautiful Soup by wesleyzhao on Tuesday June 28, 2011 @03:11PM (#2706813)
Is there any other tool that easily help with JS parsing?? This would actually be extremely useful since there are so many JavaScript generated pages.