As noted, you can run vim or emacs in a way where you can mouse where you want to go. In practice, there's usually a way to get to the place you want to be via some other command. For vi, you can usually use /search to get where you want. One of my most-used vi commands that I miss in other editors is f[char]. It moves you to the next instance of [char] on your current line. Things like f= or f( are great for bouncing through a line of source code. I wish textmate (chocolate, sublime, whatever) had a quick keyboard equivalent to that.
I use MacVim. Keyboard navigation works great for me when moving around small areas, but I use the scrollwheel and clicking all the time for moving around larger areas of a file (and for navigating NERDtree).
How does one solve this? is there a way to make the terminal accept mouse input? I've mostly only used pico/nano which I understand is similar to vi, et al. but maybe don't have a way to do this where other editors do?