It's certainly better to have source, but this seems like a matter of degree? You don't really know what's in Signal unless you compile it yourself, and/or they have reproducible builds and you verify checksums. Instead you're trusting that the source matches the binary, and probably also that someone else who knows more about crypto is reviewing the source carefully.

In the modern world we basically outsource everything, including trust and verification. An open, social process of verification can be better, though.