Except anyone on the machine can bind to a port above 1024 so if your mosh server process ever exits for any reason a compromised account could bind a backdoored version.