You probably wouldn't install all your extensions on a remote server, correct, but you get the basics like syntax highlighting and mouse support by default nowadays. Even a minimal vim install is considerably more featureful than nano.
Even without any plugins Vim has useful stuff you might need in a pinch.
Eg:
Column editing using visual mode.
Syntax highlighting
Tree-based undo features.
Fuzzy file finding in a project path.
Spell checking
Support for c tags
Line numbers
Mouse support
File navigation
On top of that you might want to learn vi for systems built with busybox.Or tools that use readline like repls and sql clients.
Speaking as a life-long vim user.