Parsing 10TB of Metadata, 26M Domain Names and 1.4M SSL Certs for $10 on AWS

Re:Parsing 10TB of Metadata, 26M Domain Names and 1.4M SSL C... by workitout on Sunday January 17, 2016 @11:17AM (#10919878)

How much did storing the data on S3 cost where you said, "However, the data is on S3" or was it there for such a transient time that it didn't cost much? Bandwidth costs in/out of S3 too?

Edit: Actually I read the S3 parts again, it sounds like the CommonCrawl project pays the S3 costs, I think, since it looks like you're using their domain data?

[ Parent | on HN ]

Re:Parsing 10TB of Metadata, 26M Domain Names and 1.4M SSL C... by jtwaleson on Sunday January 17, 2016 @11:23AM (#10919889)

The results of the Common Crawl project are hosted on AWS Public Data Sets, so it's not in my account. https://aws.amazon.com/datasets/

[ Parent | on HN ]

Re:Parsing 10TB of Metadata, 26M Domain Names and 1.4M SSL C... by workitout on Sunday January 17, 2016 @11:32AM (#10919928)

I see, without CommonCrawl paying for S3 (or AWS maybe eats that cost to help the public); this would be an expensive project.

[ Parent | on HN ]

Re:Parsing 10TB of Metadata, 26M Domain Names and 1.4M SSL C... by xrstf on Sunday January 17, 2016 @01:45PM (#10920420)

Actually, on the paged linked on your parent post, it says

> AWS is hosting the public data sets at no charge for the community

[ Parent | on HN ]