News for hackers. Stuff that matters.
Re:Scraping, cleaning, and selling big data by gtani on Wednesday May 11, 2011 @12:30PM (#2537076)
Less evil: read about how duckduckgo uses Boss and Bing APIs

http://www.gabrielweinberg.com/blog/2010/08/thoughts-on-yaho...

-----------------

and check http://www.reddit.com/r/datasets and infochimps, maybe dataset is already out there

--------------------

and ...never spider from home, you'll always screw something up: referrer HTTP header, user agent, random wait time between requests,

http://streamhacker.com/2010/10/04/perils-web-crawling/

http://news.ycombinator.com/item?id=2463058 see first comment

There are no comments on this comment yet.