I've found this to be particularly true with William Gibson, tried to listen to some but a split second lapses in attention forcing me to go back often and I gave up. And just pick up a text.

For me this difference manifests in the speed I can listen to. I typically listen to podcasts at 1.7× speed whereas I listen to (fiction) audiobooks at 1× speed. Partly this is because it lets me appreciate a good reader's work better with intonation, different voices, etc., but it's probably also what you note that they're at different information densities.

Even so, I tend to rewind when I notice I missed something, e.g. due to my attention being needed elsewhere for a bit. I mostly listen while cycling and as long as I'm outside the city there's few distractions, but when sharing the streets with others there's often distractions that are more important than listening.