News for hackers. Stuff that matters.

Zelgius

Re:Ask HN: Anyone here do Web mining/harvesting/scraping? on Monday October 19, 2009 @10:22AM (#890091)
Actually WebQL is pretty amazing. A simple program like select * from links within http://www.cnn.com to an advanced 10 page script to pull and merge multiple sources. The code is real easy to read and if you have any kind of SQL background very easy to learn. We're just always keeping our eyes out for something better/cheaper (ie free) :)

Ask HN: Anyone here do Web mining/harvesting/scraping?

I've seen a bunch of software solutions out there. WebQL lets you write SQL-like code to harvest websites. There are also GUI solutions like modenza, fetch and kapow.Does anyone here work with these kinds of solutions and what do you consider to be the best approach and why?Thanks!

( Read More... | 6 comments | 9 points | on HN )