News for hackers. Stuff that matters.

A comment on Web Scraping 101 with Python. Parent comment

Re:Web Scraping 101 with Python by wslh on Sunday March 10, 2013 @06:28PM (#5353824)
In my experience lxml.html is much better.
[ Parent | on HN ]
Re:Web Scraping 101 with Python by d0mine on Sunday March 10, 2013 @07:40PM (#5354093)
The article shows that BeautifulSoup can use lxml internally. It can also use html5lib.
[ Parent | on HN ]
Re:Web Scraping 101 with Python by thibauts on Sunday March 10, 2013 @08:38PM (#5354273)
I'll second, lxml.html is in my experience very robust and fast. I've been writing a lot of scrapers along the years and the best combination I found so far is requests / lxml.html / gevent.

It doesn't get any simpler than this IMO http://pastebin.com/hacxmAjV

[ Parent | on HN ]
Re:Web Scraping 101 with Python by danneu on Sunday March 10, 2013 @10:53PM (#5354644)
For fun here's the Ruby+Nokogiri version and my attempt at the Clojure+Enlive version (3rd day learning Clojure).

https://gist.github.com/danneu/5131596

[ Parent | on HN ]
Re:Web Scraping 101 with Python by thibauts on Monday March 11, 2013 @04:45AM (#5355562)
Pretty nice. I'd like to see how it goes in a real world scenario with concurrency, manipulation of extracted nodes and general HTTP post/auth/etc. I'm not sure about Ruby+Nokogiri but Clojure+Enlive may well be a great choice.
[ Parent | on HN ]