Every story and comment from the Hacker News API, in a layout borrowed from 2001.
A comment on Import.io – Structured Web Data Scraping. Parent comment
- Fetchbot: https://github.com/PuerkitoBio/fetchbot
Flexible, similar API to net/http (uses a Handler interface with a simple mux provided, supports middleware, etc.)
- gocrawl: https://github.com/PuerkitoBio/gocrawl
Higher-level, more framework than library.
Coupled with goquery (https://github.com/PuerkitoBio/goquery ) to scrape the dom (well, the net/html nodes), this makes custom scrapers trivial to write.
(sorry for the self-promoting comment, but this is quite on topic)
edit: polite crawlers, not scrapers.
There are no comments on this comment yet.
- Fetchbot: https://github.com/PuerkitoBio/fetchbot
Flexible, similar API to net/http (uses a Handler interface with a simple mux provided, supports middleware, etc.)
- gocrawl: https://github.com/PuerkitoBio/gocrawl
Higher-level, more framework than library.
Coupled with goquery (https://github.com/PuerkitoBio/goquery ) to scrape the dom (well, the net/html nodes), this makes custom scrapers trivial to write.
(sorry for the self-promoting comment, but this is quite on topic)
edit: polite crawlers, not scrapers.