News for hackers. Stuff that matters.

A comment on Parsing 10TB of Metadata, 26M Domain Names and 1.4M SSL Certs for $10 on AWS.

Re:Parsing 10TB of Metadata, 26M Domain Names and 1.4M SSL C... by yazriel on Sunday January 17, 2016 @10:28AM (#10919692)
Can you comment on how many additional domains you mined - compared (for example) to the 1M domains from alexa top-1M
[ Parent | on HN ]
Re:Parsing 10TB of Metadata, 26M Domain Names and 1.4M SSL C... by jtwaleson on Sunday January 17, 2016 @10:35AM (#10919714)
$ cat alexa myset myset | sort | uniq -u | wc -l

773733

0.77M of Alexa top 1M were not in my list.

$ cat alexa alexa myset | sort | uniq -u | wc -l

25842205

I mined 25,842,205 additional domain names.

[ Parent | on HN ]
Re:Parsing 10TB of Metadata, 26M Domain Names and 1.4M SSL C... by howaboutit on Sunday January 17, 2016 @03:56PM (#10921028)
Did you consider using the gTLD zone files (from the respective registries) and the ccTLD zone files found @ http://viewdns.info/data/? A much bigger initial dataset than 25M domains right there?
[ Parent | on HN ]
Re:Parsing 10TB of Metadata, 26M Domain Names and 1.4M SSL C... by jtwaleson on Sunday January 17, 2016 @04:06PM (#10921071)
No, getting access will probably take a couple of days (or in case of viewdns, more than 100$) and thereby all the fun out of the project. If you know of any other way to get the list I'd be happy to hear it though!
[ Parent | on HN ]
Re:Parsing 10TB of Metadata, 26M Domain Names and 1.4M SSL C... by adamseabrook on Sunday January 17, 2016 @04:39PM (#10921214)
http://meanpath.com/freedirectory.html

Feel free to grab a copy of our domain list. The "All domains with NS records" is the one you want. Has 191 million in it.

[ Parent | on HN ]
Re:Parsing 10TB of Metadata, 26M Domain Names and 1.4M SSL C... by jtwaleson on Tuesday January 19, 2016 @07:34AM (#10930265)
Wow! That's awesome!
[ Parent | on HN ]
Re:Parsing 10TB of Metadata, 26M Domain Names and 1.4M SSL C... by breakingcups on Monday January 18, 2016 @05:00AM (#10923530)
Amazing, thanks!
[ Parent | on HN ]
Re:Parsing 10TB of Metadata, 26M Domain Names and 1.4M SSL C... by hdmoore on Sunday January 17, 2016 @05:39PM (#10921446)
A shortcut to getting com, net, info, org, us, sk, and biz is to give premiumdrops.com $24.95/mo. You can get these for free from the TLD operators, but it takes a few weeks of snail mail (last I checked). The gTLD access via CZDAP is free, but takes a few days for approvals to process.
[ Parent | on HN ]
Re:Parsing 10TB of Metadata, 26M Domain Names and 1.4M SSL C... by adamseabrook on Sunday January 17, 2016 @05:53PM (#10921491)
https://czds.icann.org/en It has been largely automated now so you can request access to the files with one click vs having to sign and email hundreds of forms. Approval seems to be automatic for most of them.
[ Parent | on HN ]
Re:Parsing 10TB of Metadata, 26M Domain Names and 1.4M SSL C... by yazriel on Sunday January 17, 2016 @10:41AM (#10919739)
man.. the internet really is full of crappy domains...

(and yes.. now i see that you mentioned it in the article.. took me time to get there)

[ Parent | on HN ]
Re:Parsing 10TB of Metadata, 26M Domain Names and 1.4M SSL C... by hdmoore on Sunday January 17, 2016 @05:35PM (#10921424)
The Sonar FDNS set contains about 1.4 billion host names (50m+ domains). The FDNS set is seeded from TLD zones, CZDAP, PTR lookups (RDNS), SSL/TLS scans, and HTTP link extraction. It updates every two weeks: https://github.com/rapid7/sonar/wiki/Forward-DNS
[ Parent | on HN ]