You can compile Signal yourself, and install it on a rooted phone, running presumably a Linux kernel and some Android/ASOP sub-system. In that case, excepting base-band backdoors and a few other details, Google won't have access to your phone at all (assuming no Google services etc here).

OWS doesn't then allow you to use their servers for routing/discovery etc - so you need to run your own servers, and set up a different network that cannot federate with the one users of the Google Play Appstore version of Signal use.

If you do that, and install eg. the F-Droid store, you've now given another actor (the F-Droid store) access to your phone. OWS argues that in general you're less likely to manage to run a safe, patched system this way.