Re:A Look Inside Our 210TB 2012 Web Corpus
by natch on Tuesday August 13, 2013 @06:54PM
(#6208825)
How does one get set up to access the s3:// links their blog posts reference? I do realize these point to Amazon S3 buckets, but how to get at them?
[ Parent
| on HN ]
Re:A Look Inside Our 210TB 2012 Web Corpus
by WestCoastJustin on Tuesday August 13, 2013 @07:19PM
(#6208954)
Just replace 's3://' with 'https://s3.amazonaws.com/'. You can use this link [1], but it looks like most of them are returning "Access Denied", so you would likely need to login with your AWS username/password to access them.
Re:A Look Inside Our 210TB 2012 Web Corpus
by Aloisius on Tuesday August 13, 2013 @06:58PM
(#6208845)
You need an Amazon account - though the data is available for free, I think you need to specify your access key to actually fetch it.
From there you can grab the S3 command line tools (http://s3tools.org/s3cmd) or load it up from hadoop or through one of the various open source libraries (boto for instance).