upvote - python has ridiculously good (sometimes slowish) scraping libraries, unfortunately the author has written a python wrapper over a regex which completely throws any half-sane crawling out the window.

The except: and continues is enough to cause a heartattack