So you're operating with a compromised user account capable of killing your sshd on port 1234 but there's no other local escalation exploits?

Seems like something a little script or patch could fix up really easily - make sure your daemon is running on port 1234. If it's not, take whatever defensive measures you think would be effective.

Look at it a different way. If a bug was found that could kill sshd, then a local user could replace it.

Anything connecting to this replacement sshd would immediately be spotted as fraudulent due to keys.

Or instead of doing a script or a patch, you could just do the right thing in the first place and run your daemon on any of the literally over one thousand ports set aside for privileged services.

Running on a privileged port doesn't change which other hosts can connect to the privileged server though. At best it tells the client that the server is probably running as root (or has the appropriate capability bit).

If you're worried about an impostor sshd on that host then I would tend to agree that it's simply not prudent to be connecting to that server at all, let alone passing key material or credentials.