Your phone is running an APK, which is a bunch of signed code. You don't have the keys to sign such an APK yourself, but you can get tools that will tell you exactly what's inside the one you have.
I believe the Java source in GitHub is designed to be capable of a reproducible build, where you get the exact same Java binaries out as Signal's own builders did and thus you can compare that to confirm the Java code in your APK matches a specific Git checkout.
The media files (e.g. images, labels) are just straight binary copies so that's easy enough.
However there is native code to make stuff like video calls work, and when I last paid attention there was no reproducible build for that component. So you could imagine that somehow the native video call code is actually a secret backdoor or something.
Still, years ahead anything else that actually have users. The server is mostly 100% closed source. There's one open source that you can host, but it's widely believed to not be even close to the one they use.
I think only matrix is fully open and p2p.
The Signal app itself is opensource as well various pieces of the tech stack. You can audit yourself what is being sent and how their protocols work. The protocol itself has won awards due to its security and elegance.
There is a lot of good things to say about Signal and you can easily find it all. They have made some annoying or less than ideal features that are opt-out instead of opt-in but they're not sacrificing privacy for them.