You should absolutely do this, but please skip the Whisper.

The reason is that if you speak with lots of verbal fillers, that's actually an important part of how you sound to other people. It makes sense to clean up audio for a podcast, but not for your great grandchildren.

A voice cloner doesn't care that you say "um" too much. It's parsing audio for phonemes.