News for hackers. Stuff that matters.
Re:Ask HN: Anyone here do Web mining/harvesting/scraping? by cjoh on Friday October 16, 2009 @03:35PM (#886026)
We do a lot of scraping and the best we've found, really, is beautifulsoup and being good at python. There's no UI or easy tool that's going to replace that.
Re:Ask HN: Anyone here do Web mining/harvesting/scraping? by Zelgius on Monday October 19, 2009 @10:22AM (#890091)
Actually WebQL is pretty amazing. A simple program like select * from links within http://www.cnn.com to an advanced 10 page script to pull and merge multiple sources. The code is real easy to read and if you have any kind of SQL background very easy to learn. We're just always keeping our eyes out for something better/cheaper (ie free) :)