Ask HN: I created a scraper, now I think how to utilize scraped data

Posted by gorer on Tuesday September 20, 2016 @08:11AM
from the ask-hn dept.

Dear HN users.

As an exercise while learning Golang I created a bot scraping home pages of all live domains in a small country. It's more than 400 000 domains and more than 10GB of data (title and description if provided, html source of home page, pure text of home page without markup and some domain status info). Now I look for use cases how to utilize these data on the web (web site, service). If you have an idea please post a comment.

Thank you.

( 1 comment | 3 points | on HN )

Re:Ask HN: I created a scraper, now I think how to utilize s... by mtmail on Tuesday September 20, 2016 @08:22AM (#12538909)

Have a look at https://github.com/openvenues/libpostal who (one person) used a webcrawl as input to train an address parser.

[ Parent | on HN ]