News for hackers. Stuff that matters.
Re:My project: CSVGet -- Get structured data from sites as CSV by mc on Friday August 28, 2009 @10:03PM (#792662)
The real magic behind this gem is another one called Parsley, which is an astounding library for screen scraping.

http://github.com/fizx/parsley/tree/master

They've even built an entire app on top of the library: http://parselets.com/

Re:My project: CSVGet -- Get structured data from sites as CSV by tectonic on Friday August 28, 2009 @05:26PM (#792322)
Use http://selectorgadget.com to help make the selectors, too.
Re:My project: CSVGet -- Get structured data from sites as CSV by vbar on Saturday August 29, 2009 @03:47AM (#792951)
But why mess with the selectors by hand? http://search.cpan.org/~vbar/HTML-ListScraper-0.05/ can discover the structure automatically (well, sometimes :-) - some pages just aren't regular enough, but it does work on HN, for example)...
Re:My project: CSVGet -- Get structured data from sites as CSV by hadley on Friday August 28, 2009 @08:38PM (#792574)
Wow - fantastic!
Re:My project: CSVGet -- Get structured data from sites as CSV by sgrove on Friday August 28, 2009 @05:06PM (#792291)
Wow, incredibly cool. I did the same thing with collection of curl / grep / sed / awk, and it was awful. I later redid it with some python library, and then with hpricot, and most recently with scrubyt. Each step was a little bit better, but I really should have been looking towards making a more generalized solution like this.

Well done!

Re:My project: CSVGet -- Get structured data from sites as CSV by skorgu on Friday August 28, 2009 @06:25PM (#792405)
A flag to output JSON would be nifty, then you could pass it through a filter [1] and get interweb-grep on steroids.

[1] http://goessner.net/articles/JsonPath/

Re:My project: CSVGet -- Get structured data from sites as CSV by fizx on Friday August 28, 2009 @06:39PM (#792427)
Um, you got it. :) I just added a "jsonget" binary to the package.

http://gist.github.com/177304

Re:My project: CSVGet -- Get structured data from sites as CSV by skorgu on Saturday August 29, 2009 @07:44AM (#793067)
Now that's service!
Re:My project: CSVGet -- Get structured data from sites as CSV by Aschwin on Friday August 28, 2009 @06:00PM (#792375)
Oh my, this rocks. What would be even cooler is a library to use these functions. That would beat me to it, since I was thinking about a general data interface for websites too. If I could implement your effort into my code (PHP) that would be so awesome. Anyhow, nice idea though.
Re:My project: CSVGet -- Get structured data from sites as CSV by fizx on Friday August 28, 2009 @06:03PM (#792380)
http://github.com/fizx/parsley/tree/master is a C library which represents the core of the parsing functionality. A PHP binding is quite possible, and indeed, there already are bindings for Python and Ruby. In fact, I might just go look at how PHP bindings are done in general...
Re:My project: CSVGet -- Get structured data from sites as CSV by hadley on Friday August 28, 2009 @08:30PM (#792560)
This is fantastic. I do a load of data scraping of the web and this (plus selector gadget) is going to make my life sooooo much easier.