I agree, that's why signed source code releases are the safest thing you can get. Keepass has signed releases (including the source code archive) that can be checked with OpenPGP.

https://keepass.info/integrity.html

If you trust the signed source code there's no reason you shouldn't trust the signed binary - unless you have sufficient time and expertise to audit the source.

This is how I view it:

* Being open source protects against a malicious developer. Otherwise there is nothing preventing him to build the binary with a different source, and send the passwords to his own server.

* Signed code archive prevents against a compromised hosting site.

In order to get from a trusted source to a trusted binary, you have to trust the compiler and its dependencies as well, I think.