News for hackers. Stuff that matters.
Re:Import.io – Structured Web Data Scraping by thejosh on Sunday April 13, 2014 @05:26PM (#7583126)
I prefer to rely on code that doesn't rely on an API that could just vanish the next day or cost a bucket to run.
Re:Import.io – Structured Web Data Scraping by zmmmmm on Sunday April 13, 2014 @05:42PM (#7583192)
> rely on an API that could just vanish the next day

Kind of ironic that you are saying this about web scraping ...

Re:Import.io – Structured Web Data Scraping by jimktrains2 on Sunday April 13, 2014 @06:03PM (#7583285)
But then his data source is gone and what he was doing is pointless. Losing your processor of said data source while said data source is still available is frustrating.
Re:Import.io – Structured Web Data Scraping by ejstronge on Sunday April 13, 2014 @05:33PM (#7583154)
What do you use for scraping? I may have a scraping project later this year and would love recommendations.
Re:Import.io – Structured Web Data Scraping by PuerkitoBio on Sunday April 13, 2014 @06:36PM (#7583417)
I've written a couple "polite" crawlers in Go (i.e. obeys robots.txt, delays between requests to the same host).

- Fetchbot: https://github.com/PuerkitoBio/fetchbot

Flexible, similar API to net/http (uses a Handler interface with a simple mux provided, supports middleware, etc.)

- gocrawl: https://github.com/PuerkitoBio/gocrawl

Higher-level, more framework than library.

Coupled with goquery (https://github.com/PuerkitoBio/goquery ) to scrape the dom (well, the net/html nodes), this makes custom scrapers trivial to write.

(sorry for the self-promoting comment, but this is quite on topic)

edit: polite crawlers, not scrapers.

Re:Import.io – Structured Web Data Scraping by djm_ on Sunday April 13, 2014 @05:41PM (#7583191)
Scrapy gets a solid recommendation from me. http://scrapy.org/
Re:Import.io – Structured Web Data Scraping by frabcus on Sunday April 13, 2014 @06:01PM (#7583277)
We've got quite an old mailing list full of geeks hand-coding web scrapers, if you want somewhere to ask questions:

https://groups.google.com/forum/#!forum/scraperwiki

Re:Import.io – Structured Web Data Scraping by egeozcan on Sunday April 13, 2014 @05:57PM (#7583252)
I use custom node.js scripts with these libraries:

* request - https://github.com/mikeal/request

* async - https://github.com/caolan/async

* cheerio - https://github.com/cheeriojs/cheerio

* nedb - https://github.com/louischatriot/nedb