News for hackers. Stuff that matters.
Re:Import.io – Structured Web Data Scraping by ejstronge on Sunday April 13, 2014 @05:33PM (#7583154)
What do you use for scraping? I may have a scraping project later this year and would love recommendations.
Re:Import.io – Structured Web Data Scraping by PuerkitoBio on Sunday April 13, 2014 @06:36PM (#7583417)
I've written a couple "polite" crawlers in Go (i.e. obeys robots.txt, delays between requests to the same host).

- Fetchbot: https://github.com/PuerkitoBio/fetchbot

Flexible, similar API to net/http (uses a Handler interface with a simple mux provided, supports middleware, etc.)

- gocrawl: https://github.com/PuerkitoBio/gocrawl

Higher-level, more framework than library.

Coupled with goquery (https://github.com/PuerkitoBio/goquery ) to scrape the dom (well, the net/html nodes), this makes custom scrapers trivial to write.

(sorry for the self-promoting comment, but this is quite on topic)

edit: polite crawlers, not scrapers.

Re:Import.io – Structured Web Data Scraping by djm_ on Sunday April 13, 2014 @05:41PM (#7583191)
Scrapy gets a solid recommendation from me. http://scrapy.org/
Re:Import.io – Structured Web Data Scraping by frabcus on Sunday April 13, 2014 @06:01PM (#7583277)
We've got quite an old mailing list full of geeks hand-coding web scrapers, if you want somewhere to ask questions:

https://groups.google.com/forum/#!forum/scraperwiki

Re:Import.io – Structured Web Data Scraping by egeozcan on Sunday April 13, 2014 @05:57PM (#7583252)
I use custom node.js scripts with these libraries:

* request - https://github.com/mikeal/request

* async - https://github.com/caolan/async

* cheerio - https://github.com/cheeriojs/cheerio

* nedb - https://github.com/louischatriot/nedb