Dead simple Python crawler for extracting structured data from a website to CSV

Re:Dead simple Python crawler for extracting structured data... by orf on Wednesday January 06, 2016 @08:25PM (#10855322)

The author has just decided to roll their own half baked script for some reason, there are plenty of libraries to help. Python is good for scraping, I released my own library recently[1] that I think hits a sweet spot but there are many others.

Python and Ruby have their places, I wouldnt say one is clearly better than the other.

1. http://tomforb.es/scraping-websites-with-cyborg

[ Parent | on HN ]

Re:Dead simple Python crawler for extracting structured data... by rangeva on Thursday January 07, 2016 @01:41AM (#10856386)

The script is intended to be a copy/paste script, with no imports and just "Fill in the blanks" parameters. There A LOT of other options that are much better, but require more setup: http://scrapy.org/ https://pypi.python.org/pypi/spider.py

[ Parent | on HN ]

Re:Dead simple Python crawler for extracting structured data... by aerioux on Thursday January 07, 2016 @02:53AM (#10856595)

upvote - python has ridiculously good (sometimes slowish) scraping libraries, unfortunately the author has written a python wrapper over a regex which completely throws any half-sane crawling out the window.

The except: and continues is enough to cause a heartattack

[ Parent | on HN ]