A number of the sponsoring orgs will grant you access to their zonefiles if you're willing to do a little paperwork (HelloFax is awesome) and have a valid reason for access. Granted it's not the complete list of registered domains (those without nameservers on file won't show up), but it's pretty close.

The list of sponsoring orgs for each TLD is at: http://www.iana.org/domains/root/db

Extracing such a list from the generated index only takes a small script and a few hours to download the 200+ GB index file. Which is a lot less than the slightly bigger script and the months/years to download and process 80+ TB of arc files that it previously would have taken to extract all domains.

Anyways, if you want a copy of domains from the index file just send me a mail to the address in my profile.

Where is the index file located in the s3 bucket?

s3://aws-publicdatasets/common-crawl/projects/url-index/url-index.1356128792

thank you kindly!

You can get the top million sites from Quantcast in a download for free. Alexa used to also offer something similar, but I don't see it on their site any longer (they try to sell their data through aws, so that's probably why).

You can find the Alexa top 1 million domain list at http://www.alexa.com/topsites (it's on the right hand side).

Can someone dig that up and link it please

That would be great. Out of curiosity, mind describing what you'd use such a list for?

Usually as a starting point for a crawler when checking statistics about the web.

Crawling is much faster if you don't have to "spider" all the links and check if you've already visited them or not.

With a big enough list, you can just iterate over those domains. (average number of links on a website, how often does javascript framework x vs y get used, how many sites have an HTML5 doctype yet, ...)