Didn't consider that the mode can be set from environment variable, then it can just be automatically set from my ssh command, thanks for the tip!

Searching back in the buffer in emacs mode is the one thing I've been keep having to look up