News for hackers. Stuff that matters.
Re:How to crawl a quarter billion webpages in 40 hours by alexbardas on Saturday August 11, 2012 @07:23AM (#4369883)
Yes, but it's hard to tell how complete and updated that subset is. There is (was until few weeks ago apparently) also http://www.imdbapi.com/ which was retrieving their data by crawling. Unfortunately, it was shut down.

There are no comments on this comment yet.