News for hackers. Stuff that matters.
Re:Free 5 Billion Page Web Index Now Available from Common C... by Aloisius on Tuesday November 08, 2011 @03:38PM (#3212683)
I imagine they don't use bittorrent because it is both very large (TBs) and changes frequently.

With S3, you could boot up a bunch of Hadoop processes, pull it (without incurring any bandwidth costs I believe), process it and dump out whatever you want.

There are no comments on this comment yet.