Common Crawl claims to have 82% of the tokens used to train GPT-3, and it's available to anyone.

Add all the downloadable material at archive.org and you've got a formidable corpus.

https://commoncrawl.org/