It's not about the software, it's about the distribution of the software. Signal distributes through Google as man-in-the-middle. So if you trust Google, you're OK. But if you're using Signal because you don't trust Google, it doesn't make sense.

In case you don't trust Google I really suggest OpenSSH or WireGuard, to establish a secure tunnel; and then run IRC, Jabber, SIP, or mumble (as needed) over the tunnel.