Triv.io donates URL index to Common Crawl

Re:Triv.io donates URL index to Common Crawl by srobertson on Tuesday January 08, 2013 @02:42PM (#5028029)

Let me double check that for you. Were you using a valid aws-id and secret?

[ Parent | on HN ]

Re:Triv.io donates URL index to Common Crawl by soult on Tuesday January 08, 2013 @02:54PM (#5028079)

No, usually you can download them without sending any aws-id if they are on the Public Datasets S3 bucket, e.g.

    wget https://s3.amazonaws.com/aws-publicdatasets/common-crawl/crawl-001/2008/06/19/0/1213886083018_0.arc.gz

[ Parent | on HN ]