Anyone know if something like this exists?
There are, e.g., search engines that search for strings in the page source, e.g., to detect use of certain Javascript files. These are slow and not free.
Requesting zone files from the registry was the traditional method. ICANN tries to require registries to provide them to the public, with limited success. Downloading com.zone/net.zone from Verisign should be relatively straightforward (not sure if edu.zone is available anymore). However with gTLDs there are hundreds of registries now, with potentially hundreds of different rules on zone file access; some registries like ccTLDs never had zone file access programs. Even registries that seem like they would be easy to deal with can have silly restrictions, e.g., the .org registry used to have a requirement that the requester needed to have stable IP address.
Sometimes larger TLDs take a bit longer to respond to requests, whereas some others automatically accept all requests.
https://www.icann.org/resources/pages/czds-2014-03-03-en https://czds.icann.org/home
Arguably a web comprising a large number small, diverse websites, where each user may be visiting a variety of different websites, is less suitable for advertising than one where all web users are funneled through a few large websites that survive by selling online ad services, like Google. It stands to reason that those large, online ad services sites would have little interest in showing users an undiscovered portion of the web. They want users to congregate on "popular" sites. Good for advertising.
OTOH, using zone files instead of a search engine, social media or news aggregator site in the online ads (or VC) business, one can see all websites that have registered an ICANN domain name. No filters. No advertising-related algorithms. Popularity is irrelevant. The user determines relevance, not a third party.