You don't have to click down multiple times. If you want to go somewhere specific there are too many ways to list tbh but they're all faster than arrow keys or mouse clicks. I'm a big fan of the EasyMotion https://github.com/easymotion/vim-easymotion approach myself but the built-in search functionality + tag navigation is also good.