The reason to not having to move your fingers/arms much is not about time. You just stay in the zone longer, you have less possibilities of distractions... It is about comfort and not disturbing the flow.
It's the same with music (wether is jazz or blues at least). Good intrumentists just don't move their hands much, lest their instrument goes in the way of their improvisation. 60% of the time, our job is quite boring, so make sure you really enjoy the 40% that's left.
Is that really something you use all the time? I tried to switch from emacs to vscode in 2017, but in practice the minimap didn't do that much for me. Nor did the graphical file view to the left. The emacs experience is just too fast and good, even though it means I need to carry my customizations with me everywhere to get those intellisense popups and spelling squigglies.
VSCode is fast - certainly as fast as my brain. Atom was crap IME but MS have done a really good job regarding performance for VSCode. Start-up time is a bit slow, but you only do that once when you reboot every few days/weeks/months
So apart from lacking the visual feedback of scrollbars etc, sounds like emacs is the same.
FWIW it is 2 key strokes in VSCode (ctrl-g, then the line number or -n to index back from the end of the file) out of the box. Ctrl-p takes you to any fuzzy-matched file in your workspace.
My point is that VSCode has a lot of shortcuts too - you don't need to use the mouse if you really do need those extra couple of seconds time savings in your 8 hour work day in order to meet your deadlines. But on the plus side you can use they keyboard or the mouse as you see fit - best of both worlds.
If you really are working in such a sweatshop where saving a couple of seconds a day is something you need to think about, then you have my pitty and condolences! At least for me, not every single waking second of my work day is spent hammering out code - I am probably only actively typing in new code perhaps 20-25% of the time max with the rest going on reading existing code, stepping through a debugger, dealing with tickets/bugs, running tests, doing emails, meetings etc etc.
For me, something like VSCode offers some nice visual benefits, e.g. the minimap is super useful, subtle indentation lines, line-length rulers, intellisense popups near where I am actually looking, spelling squigglies etc. I would not be surprised to learn that vi/Emacs can do the same, but if you are adding all that to a text mode editor, why bother and not just use a modern visual editor and just be done with it? Just muscle memory?