it's possible to accept/drop(/reject) traffic from specified port if the listening process owner is this-or-that (see nftables "owner" module), but it can not prevent other users to bind on that port, only prevent them to actually communicate on it.