I listen to ~ 40 hours of podcasts per week, so I'm probably on the opposite end of the spectrum from you.

If you don't understand the appeal of listening to audio podcasts, I don't think there's anything I could say to convince you otherwise. The experience is exactly how you describe it, so I don't think you're missing anything.

I find myself similarly confused whenever someone posts a link here to some video and there's a raft of comments complaining about how video is so inefficient, and how people should just post everything as transcripts.