I’ve done something similar here https://github.com/mcdallas/summarize it feeds an audio file to whisper and then summarizes the transcript. You can easily wrap it with yt-dlp to download the audio portion of a video