Coming from a similar background I found that it was super helpful to create my own keybinds for stuff like navigating forward/backward through tabs, toggling the sidebar etc. Before I did that I felt similarly that it was too mouse dependent.

The VSCode Vim extension has a `gh` key sequence for showing the hover info you normally have to use the mouse for that made switching much easier.