I think it's not as universal as you believe. I personally use closed captioning as a crutch when I'm half-ass watching something, because if my mind wanders and I miss hearing a line I can usually just read it without having to pause and rewind.

If I'm actually fully immersed in the movie or whatever, and I can understand the dialogue, then I don't need the captions and I leave them off.