You can audit binaries. So definitely not "You can't full stop". Most people are in no position to audit all the source code they use, just like most technical people are in no position to audit binaries they use.

(And I'm putting aside the whole issue of backdoors in hardware, compilers, etc.)