Parsing 10TB of Metadata, 26M Domain Names and 1.4M SSL Certs for $10 on AWS

Re:Parsing 10TB of Metadata, 26M Domain Names and 1.4M SSL C... by jtwaleson on Sunday January 17, 2016 @11:23AM (#10919889)

The results of the Common Crawl project are hosted on AWS Public Data Sets, so it's not in my account. https://aws.amazon.com/datasets/

[ Parent | on HN ]

Re:Parsing 10TB of Metadata, 26M Domain Names and 1.4M SSL C... by workitout on Sunday January 17, 2016 @11:32AM (#10919928)

I see, without CommonCrawl paying for S3 (or AWS maybe eats that cost to help the public); this would be an expensive project.

[ Parent | on HN ]

Re:Parsing 10TB of Metadata, 26M Domain Names and 1.4M SSL C... by xrstf on Sunday January 17, 2016 @01:45PM (#10920420)

Actually, on the paged linked on your parent post, it says

> AWS is hosting the public data sets at no charge for the community

[ Parent | on HN ]