Yes, that's in fact the source of the commonly cited LibriSpeech dataset. It used publicly available recordings from the LibriVox project and has done some cleaning steps as well as alignment of the transcript with the audio to arrive at 1000 hours of cleaned up audio.
The M-AILABS dataset uses LibriVox as well, among other sources, to arrive at 1000 hours of cleaned up data in various european languages.
Overall there's still a large untapped potential in LibriVox data: https://gist.github.com/est31/1e195c55fab8f95a72393db1519da1...
But you'll have to give up properties like gender balancing in your dataset that then has to be counteracted during the learning process, e.g. by having gender-custom learning rates.
I think the next step here needs to be auto-finding librivox source books that aren’t in e.g. Gutenberg. I only auto-found books for 75% of English. Only found books for 20% of German. I don’t actually know where to look beyond Gutenberg for books that don’t have easily linked text. Common crawl? Librivox forums?
(Or you can take Facebook’s semi-supervised approach and generate a transcript from thin air instead, but imo that will work better for English than languages that don’t have good baseline models yet)
I dare you to try any STT in more "niche" languages or dialects. US English should not be considered the be-all-end-all indicator for progress in this area of research.
http://news.bbc.co.uk/1/hi/5244738.stm
What I thought amusing was the it was pretty clear that the people shown talking are trying really hard to talk "properly" - their real accents would probably be much broader.
While for copyright reasons this dataset likely can't be released publicly, the method to generate it from materials should be possible to release so that IF you have the source, you can get the annotated audio.