News for hackers. Stuff that matters.

A comment on How to crawl a quarter billion webpages in 40 hours. Parent comment

Re:How to crawl a quarter billion webpages in 40 hours by WalterGR on Saturday August 11, 2012 @06:47AM (#4369861)
I managed to crawl and store (in mongo) more than 300k movies from IMDB in just a few hours

Did you know that IMDB makes a subset of their data publicly available? http://www.imdb.com/interfaces/

[ Parent | on HN ]
Re:How to crawl a quarter billion webpages in 40 hours by alexbardas on Saturday August 11, 2012 @07:23AM (#4369883)
Yes, but it's hard to tell how complete and updated that subset is. There is (was until few weeks ago apparently) also http://www.imdbapi.com/ which was retrieving their data by crawling. Unfortunately, it was shut down.
[ Parent | on HN ]