I do one ear when I'm walking or working. I do both if I'm at the gym.
for podcasts, if you use a podcast app like Overcast that supports playback speed adjustments and silence skipping you might find a speed that keeps your attention while still retaining the important stuff.
I wonder if any audiobook apps support that kind of speedup feature?
When I try to listen to podcasts at home my mind usually ends up drifting. And I can't stand having earbuds with music or anything else in my ears when walking. Separates me from my environment too much.