Couldn't you just download subtitles for old movies and train using those?

There aren't many old movies (in the public domain) that are subtitled and/or have good audio... However, in the U.S., we have a huge amount of government audio/video that is captioned. Here's an example of using youtube-dl to download one of President Obama's speeches and extract the closed captions (which I believe are produced by the government) into its own VTT file:

https://asciinema.org/a/93ihv5er83mrpihh9i5m3s9o5

Congress has video for all of its sessions and it is transcribed. So does the Supreme Court (though not timestamped).

Doesn't even need to be old movies. Certain types of video content in the US is legally required to have subtitles(e.g. a lot of youtube content). You could programmatically download them and use that as your training set. And, since it is a transformative work, you can train your models even on copyrighted works freely.

Much of the YouTube content has auto-generated subtitles, i.e. Google is running their speech-recognition software on the audio stream and then using that to caption the video. If you used that as your training set, you're effectively training on the output of an AI. Which is kind of a clever way to get information from Google to your open-source library, but will necessarily be lower-fidelity than just using the Google API directly.

In the US, if it's ever been played out on broadcast TV then it must have Closed Captions.

This is enforced by the FCC [0], but as more and more "internet" content gets consumed I imagine the same regulations will eventually come, at which point you've got a fantastic training set.

0: https://www.fcc.gov/node/23883

That's actually a pretty good idea. Lyrics for rap music might also be a good training data set. It'd bias strongly toward English, though, and particularly American English. I suspect the size of the resulting data set is also quite a bit smaller than what Google has.