Re:Free 5 Billion Page Web Index Now Available from Common C...
by Aloisius on Tuesday November 08, 2011 @03:38PM
(#3212683)
I imagine they don't use bittorrent because it is both very large (TBs) and changes frequently.
With S3, you could boot up a bunch of Hadoop processes, pull it (without incurring any bandwidth costs I believe), process it and dump out whatever you want.
With S3, you could boot up a bunch of Hadoop processes, pull it (without incurring any bandwidth costs I believe), process it and dump out whatever you want.