Think about what you actually need to start a video. Maybe a dozen MB?

After that, you can plunge into colder storages and warm things up as you stream. Additionally, if you need longer to 'defrost' things, just cache a few more MB at the front. Cheat a bit by assuming 480p to start with if you need to; even less to store.

There is also location location location.

Maybe Google holds your content in 7 data centers round the world (~1 per continent for planned maintenance + latency + reduced oceanic fiber usage).

But with old rarely streamed content they might cut that down to just 3.

Speed/latency doesn't tell you much, because it's all on a hard drive somewhere.

The question is whether YT is serving up the one (redundantly-backed storage) copy they have of your almost-never-watched video, or whether it's serving it up from one of 1,000+ copies it's made across the globe for currently popular videos.

But they are tiny 480p files yeah?

That doesn't match my experience. I have some unlisted videos that I or a small handful of friends might go back and watch once a year, and it takes several seconds of loading before they start playing. It's very noticeably different from the near-instant loading of most videos I watch.