But if I have firewall policies that allow connections only to specific services I need a destination address and port (yes, some firewalls allow host names but there's drawbacks to that)

> IP addresses aren't authenticated, they can be spoofed

For anything bidirectional you'd need the client to have a route back to you for that address, which would require you compromising some routers and advertising it via BGP etc.

You can spoof addresses all you want but it will generally not do much for a stateful protocol