domainname IP-address # timestamp
The source of domains is usually HTML, XML sitemaps or proxy log but it can be anything. I have small utilties I wrote to extract URLs and domains from any input. For example, I retrieve data for all the domains that appear on HN, excluding the ones in comments. I see there are close to 68K domains from the past 6-9 months.
(a) HTTP/1.1 pipelining and DoH servers for bulk DNS data
(b) localhost-bound authoritative DNS server for redirecting all domain names to forward proxy IP address
(c) localhost-bound forward proxy that stores domain-to-IP address mappings in memory
NB. Without modification, this setup is probably not going to work well for people who like to use graphical web browsers for everything. For recreational web use I am not using a graphical browser that auto-fetches resources. I am using a text-only browser. Thus, I only need DNS data for the domain names in the URLs I visit, not for ones found in web pages pointing to fonts, CSS, images, Javascript, and so forth. This makes the DNS data retrieval process very straightforward: one domain name for each URL. In the times I have tested recreational browsing using a graphical browser, it can be surprising how many web pages are quite readable without using resources from other domains, but for folks that want to view myriad web pages that are comprised of resources from a variety of hosts requiring numerous domain name lookups, bulk DNS data retrieval will require more work. In any event, there are many, many possible variations of a setup like this and the one I use is suited for me; I am not fond of the large graphical browsers and I rely on the proxy for a number of tasks. Other variations might suit other users' personal preferences.
Hope that helps.
By not using recursive DNS, including open resolvers or other third party DNS providers, I avoid this problem. I do not need to maintain a list of domains to block,^1 I only need to maintain a list of domains that I actually want to access. I do not even make remote DNS lookups because generally I already have the DNS data stored (gathered in bulk beforehand), so the web becomes faster, not slower. One of the requirements for the web to suck the way it does, and for so-called "tech" companies to proliferate by collecting data and selling ad services, is that the web browser and webpages must be give free reign to lookup any domain name. IME over nearly 20 years of controlling DNS, restricting that ability makes the web instantly readable, even so-called "dumpster fire" webpages.
1. AFAICT no one actually does this anyway. They delegate the task to third parties, "blocklist" maintainers.