https://linux.die.net/man/8/iptables
--pid-owner processid
Matches if the packet was created by a process with the given process id.
But was disappointed by the note:> NOTE: pid, sid and command matching are broken on SMP
That's a really outdated version of the man page. The current one is: https://ipset.netfilter.org/iptables-extensions.man.html
Probably the closest thing these days is to use the cgroup match extension, together with something that makes sure separate apps are moved to their own cgroup. But that cgroup would be under a separate parent cgroup for each user so it's not ideal.
I expect NFQUEUE or some EBPF magic is a better way to do things - you don't want to be constantly adding/removing per-process iptables rules.
The iptables rules can be converted directly to nft with iptables-translate and the syntax is more simple.
Firejail [2] and Bubblewrap [3] can put limits on application capabilities and/or take away network access.
A round-about and heavy handed way to manage communications by PID would be to isolation applications in their own VM or container and map the VM or container to a unique user or group.
[1] - https://ipset.netfilter.org/iptables-extensions.man.html
But does it have any sense? Usually you block inbound connections, allowing only certain services. If a rogue process starts listening on a local port, you could display a warning alert, and as inbound connections are already blocked you'd be safe (as long as you trust netfilter...) and you wouldn't need to ask the user to perform an action.