As someone who has entirely replaced listening to music with listening to podcasts and audiobooks your post confuses me. I experience no more cognitive load than I would if I was having a conversation on the subject. If I have the mental bandwidth to listen to music while I’m working on a task then I have the bandwidth to listen to a podcast too.

Your last paragraph also seems to be painting people who listen to podcasts into a very narrow corner. You can find podcasts on everything from history and science to comedy and games. Personally one of my favorite podcasts right now is a comedy focused dnd campaign that I throw on when I drive to work.