Debian and others realize that this is a problem, and you absolutely shouldn't trust anyone at their word. There's an ongoing effort at Debian to make all builds reproducible so that you can verify that they were built from the sources they provide for inspection. This should be completed soon.

For the situation I'm talking about where absolute confidentiality is required, the right way to do it would absolutely be auditing all code and compiling from source, assuming you're working in an organization that has that kind of capacity.

But for most cases, just dealing with an organization you trust is enough, whether they provide source or not.