Google has a lot of scanned/ocr'd books that aren't available in public data sets.

I wouldn't be surprised if they used anonymized gmail data though. Even if they removed all proper nouns it'd probably provide a lot of extra tokens for training.