Privileged ports are a permission applied to bind() to a sockaddr of type AF_INET.
Privileges are also checked when bind()ing to type AF_UNIX -- aka a unix domain socket -- aka a path to a file in your filesystem.
Privileges are also checked when open()ing a local file.
It is entirely reasonable to rely on filesystem privileges to control access to regular files like /etc/shadow, or /etc/ssh/ssh_host_ed25519_key.
It is entirely reasonable to rely on filesystem privileges to control access to unix domain sockets.
It is entirely reasonable to rely on interface privileges to control access to inet sockets.
The commonality in all of the above scenarios is that it is reasonable to trust the local system regarding its own access controls. There is no such practical concept as a MiTM between a process and open() to a local file, nor is there such thing as MiTM between a process and a bind() to a local resource.
In fact, this local trust is a required component to implement a key based system (note: the need of sshd to trust file permissions to store a host key)
Clients should be configured correctly to never ask. But many won't be and that's out of your control.