Have you ever used emacs? If not theres a mode called ace-jump, which I'm pretty sure beats every interaction with a mouse to get to a certain point within a document. I'd wager that, if we'd start at the same time, I'd be at that point before you actually touch the mouse with the hand that you just lifted. Honestly, it's worth checking out. There is a lot of things that emacs and vim are brilliant at, they had tens of years to figure things out after all...
There are some interactions where switching to the mouse and back is faster; things like drawing and navigation through irregular shapes (I.e. “not grids or lists”).
Whether that difference is important to you or not is far more about your specific circumstances than anything else. Comfort is important also.
Of course they're not on the keyboard while I'm exclusively reading. But while I'm in vim, my hands certainly are on the keyboard nearly at all times, resting on the home row.
This is similar to the way a mathematician will often ‘read’ math with a pencil and paper, or a musician will ‘read’ music with an instrument in hand.
When I’m reading prose, I often don’t have either a keyboard or a mouse, but that’s a very different operation (uses different parts of the brain, etc).
I'm also not really convinced about the "hands on the keyboard" argument... What percentage of the time that you work are your hands actually on the keyboard and ready to input? Are they on the keyboard right now? Because that seems like an uncomfortable way to sit at a workstation. I also don't think it takes very much time to switch between the input devices unless maybe you don't have practice with that workflow.