I love CT logs! I use them to discover subdomains for my hosts file block list [1]. While it can't expand on domains that use wildcard certs - or no certs at all - it's better then nothing since hosts files don't support wildcard blocking.

Some things I've learned while working with them:

* CertSpotter [2] is a fantastic CT client written in Go that supports pattern matching. I've been running it locally with `.` match pattern and so far have a 4 gig file of unique domain names. I'm excited to see the end result once it catches up to current time.

* https://crt.sh/ is a great website to search CT logs and supports wildcards. It's currently the workhorse behind my hosts project, but I hope to remove them as a dependency once my own domain list is caught up to present day

* It looks like OP's tool is just a thin client for entrust API [3] and is not actually downloading logs directly - which isn't clear in the article. It made more since once I figured that out because these logs are huge and go back years.

[1] https://github.com/lightswitch05/hosts

[2] https://github.com/SSLMate/certspotter

[3] https://www.entrust.com/ct-search/

I write security reports for websites, and I use CT to inform the website owner if there are unused certificates for the given domain. Usually the customer is quite surprised that this information publicly available.

But in 99% of the cases it's not so much a security problem. For bug hunters it may be usable as unlisted subdomains have less exposure, so they may be the first to scan it for bugs. It is still a concern for the website owner though, because they don't want the world to know about a new product or experiment they are running.

General advice: don't obtain certificates for a subdomain until you are ready to tell the world about it.

Or even better don’t register CNAMES or A records for your sub domain until you’re ready to tell the world. The cert is meaningless if there’s nowhere for the traffic to route.

> The cert is meaningless if there’s nowhere for the traffic to route.

The cert has a meaning: it reveals your intent to do something with it.

I.e. if apple was to buy a cert for car.apple.com before they announce a car, that could be bad for them.

That’s fair point for giving intent if there’s a human facing name for the DNS entry. I was referring to the security implications of having a public endpoint exposed, or more accurately not being exposed because there’s no way to route traffic to it.

I love that Symantec has a bug even in their basic “search the log” logic.

The minor note that his ISP rate limited DNS to 36 or 50 response batches stood out to me. I don't understand why they'd want to do that or how that benefits them.

Is this the sort of targeted traffic shaping net neutrality would prohibit?

DNS (used to be?) is a great amplifier for DDOS attacks. Because the 'source' address is user supplied, and there are DNS responses that are much larger than the requests. So you can use 1Mb/s of data to send DNS queries with your targets IP as the source address to get e.g. a 10Mb/s stream of data to your target.

Maybe the rate-limiting is an attempt to subvert being used in this kind of DDOS.

He was [likely] using his ISP's or Other providers resolvers, Large bursts of queries are very annoying to deal with, and are very rarely legit ( and more software going wrong )

It's very common to rate limit querying down to some low number to protect the rest of the customers using it

Likely a defense against DDOS.