When you get right down to it the issue is port numbers just plain don't work for this. SELinux had some concept of sending SELinux security contexts over the network so you could get positive claims about what you were talking to, but it was never developed into a full system AFAIK.

Which is ultimately what you want: TLS "I'm securely talking to the controller of this domain name" and then a secondary "within that controllers namespace, I'm definitely talking to the service I think I am".