Re:Import.io – Structured Web Data Scraping
by thejosh on Sunday April 13, 2014 @05:26PM
(#7583126)
I prefer to rely on code that doesn't rely on an API that could just vanish the next day or cost a bucket to run.
[ Parent
| on HN ]
> rely on an API that could just vanish the next day
Kind of ironic that you are saying this about web scraping ...
Re:Import.io – Structured Web Data Scraping
by jimktrains2 on Sunday April 13, 2014 @06:03PM
(#7583285)
But then his data source is gone and what he was doing is pointless. Losing your processor of said data source while said data source is still available is frustrating.
Re:Import.io – Structured Web Data Scraping
by ejstronge on Sunday April 13, 2014 @05:33PM
(#7583154)
What do you use for scraping? I may have a scraping project later this year and would love recommendations.
Re:Import.io – Structured Web Data Scraping
by PuerkitoBio on Sunday April 13, 2014 @06:36PM
(#7583417)
I've written a couple "polite" crawlers in Go (i.e. obeys robots.txt, delays between requests to the same host).
- Fetchbot: https://github.com/PuerkitoBio/fetchbot
Flexible, similar API to net/http (uses a Handler interface with a simple mux provided, supports middleware, etc.)
- gocrawl: https://github.com/PuerkitoBio/gocrawl
Higher-level, more framework than library.
Coupled with goquery (https://github.com/PuerkitoBio/goquery ) to scrape the dom (well, the net/html nodes), this makes custom scrapers trivial to write.
(sorry for the self-promoting comment, but this is quite on topic)
edit: polite crawlers, not scrapers.
Scrapy gets a solid recommendation from me. http://scrapy.org/
We've got quite an old mailing list full of geeks hand-coding web scrapers, if you want somewhere to ask questions:
Re:Import.io – Structured Web Data Scraping
by egeozcan on Sunday April 13, 2014 @05:57PM
(#7583252)
I use custom node.js scripts with these libraries:
* request - https://github.com/mikeal/request
* async - https://github.com/caolan/async
* cheerio - https://github.com/cheeriojs/cheerio