Sure, if you use only the OpenSSH client and other software respecting its command line option paradigms or configuration files. But there's tons of other things connecting to SSH.

And even only in the OpenSSH universe, I find it quite annoying having to remember whether the `-p` can go after the hostname or has to appear before, whether SCP uses `-p` or `-P`, whether `ssh-copy-id` supports one or the other etc.

There are some protocols I wouldn't necessarily run on their default port and publicly accessible, but SSH is really not one of them, also given that sshd has been specifically hardened for that adversarial use case.

You can rsync, scp and what not using your .ssh/config alias.

    scp myfile.txt myhost.tld:~/foo