there are builds (macvim, etc) that are good with mice, if it bugs you too much. obviously, the terminal-oriented versions wouldn't do great on this front (as far as i'm aware). i find the mouse mostly nice for scrolling (which i believe does work for me for neovim on arch linux in dwm, but ymmv) at this point, but, yes, for the most part that feeling subsides when you really start to grok the navigation in normal mode (by paragraph, etc.)