Web Scraping 101 with Python

Re:Web Scraping 101 with Python by thibauts on Sunday March 10, 2013 @08:38PM (#5354273)

I'll second, lxml.html is in my experience very robust and fast. I've been writing a lot of scrapers along the years and the best combination I found so far is requests / lxml.html / gevent.

It doesn't get any simpler than this IMO http://pastebin.com/hacxmAjV

[ Parent | on HN ]

Re:Web Scraping 101 with Python by danneu on Sunday March 10, 2013 @10:53PM (#5354644)

For fun here's the Ruby+Nokogiri version and my attempt at the Clojure+Enlive version (3rd day learning Clojure).

https://gist.github.com/danneu/5131596

[ Parent | on HN ]

Re:Web Scraping 101 with Python by thibauts on Monday March 11, 2013 @04:45AM (#5355562)

Pretty nice. I'd like to see how it goes in a real world scenario with concurrency, manipulation of extracted nodes and general HTTP post/auth/etc. I'm not sure about Ruby+Nokogiri but Clojure+Enlive may well be a great choice.

[ Parent | on HN ]