Not always. TrueCrypt was open source, and yet a huge number of people were worried that the distributed binaries could be backdoor-ed in some way. See this article about someone trying to create a reproducible binary of TrueCrypt.[0]

[0]https://madiba.encs.concordia.ca/~x_decarn/truecrypt-binarie...