Your brain can't hold the context long enough to go to the required level of abstraction, while you're multitasking (may be walking or something deeply automated doesn't count.
But mathematicians can talk to each other about arbitrarily abstract concepts, as long as they have enough shared background, and they don't (always) need a blackboard to do it.
Conversely, you can have conceptually very simple things that are basically impossible to follow just by listening, like multiplying two nine-digit numbers or following one of Euclid's proofs in plane geometry. The difficulty isn't about abstraction, but how many things you have to hold in working memory
If it's 'serious' and worthwhile, I often don't even have the bandwidth to keep up with the lecture or book. Why spend my time on anything less?
Regardless, listening to something intently and doing mechanical actions are not exclusive.