Re:Scrapy Tips from the Pros
by bbayer on Wednesday January 20, 2016 @02:43PM
(#10940769)
I love scraping web and produce structured data from web pages. The only downside of using XPath or similar extracting approach is necessity of constant maintenance. If I have enough knowledge about machine learning, I would like to write a framework that analysis similar pages and finds structure of data without giving which parts of page should be extracted.
[ Parent
| on HN ]
Maybe you should give a try at Portia (http://scrapinghub.com/portia/). It does exactly what you mean.
You may also be interested in this library: https://github.com/scrapy/scrapely