News for hackers. Stuff that matters.
Re:Free 5 Billion Page Web Index Now Available from Common C... by rb2k_ on Tuesday November 08, 2011 @02:46PM (#3212441)
Oh nice. I've been doing a lot of crawling myself (http://blog.marc-seeger.de/2010/12/09/my-thesis-building-blo...) and I'd love to get my hands on this data. I hope they'll segment their data a bit further.

I personally would LOVE to have a simple list of the domainnames themselves without all of the connections and documents.

Also: Why not just use bittorrent to distribute it?

Re:Free 5 Billion Page Web Index Now Available from Common C... by Aloisius on Tuesday November 08, 2011 @03:38PM (#3212683)
I imagine they don't use bittorrent because it is both very large (TBs) and changes frequently.

With S3, you could boot up a bunch of Hadoop processes, pull it (without incurring any bandwidth costs I believe), process it and dump out whatever you want.