The options are, either you do peer-to-peer and so your peer must learn the IP address they can reach you on, or Signal sits in the middle of the traffic relaying between the parties.

This trades two different privacy risks, would you prefer that a hypothetical adversary who has successfully seized control of Signal can see which IP addresses are communicating or would you prefer if people you accept realtime calls from or make calls to learn your IP address?

You get to pick which you prefer in the Signal app preferences. [Edited to add: Specifically, if either of you insists on having Signal relay the traffic, then that's what has to happen, otherwise it is peer-to-peer.]

As with anything else involving IP addresses, you could choose to go via Tor, with all the consequences of that.