[1] https://scrapy.org/ [2] https://pypi.python.org/pypi/beautifulsoup4
To second another comment, I am also working on scraping but for articles. In this particular example I am trying to figure out how to get the scraper to start on the first headline all the way down to the bottom, and not touch anything else. Any tips for how to get it to find the first heading on a page?
I usually use lynx, which strips down the html and only outputs the text. News site tend to have the same structure for every article, so after passing the html to lynx, the headline for a news site article will probably be around line 5-6.
This is something I have tried to do with natural languages in the past but getting good, error free, unambiguous questions out of an algorithm seems to be really difficult.
It worked by scraping the top 1000 Bing results and then scraping several levels deep from each of the results to generate a "map" of knowledge. This in turn could be used to ask the user questions, generated fill-in-the-blanks, matchups & whatever else you could think up.
It worked after the five-day hackathon, which was truly surprising, but unfortunately, it used too much computing power so we never released it to the world.