> If I can compile audited code from source myself, without any backdoors, then I can be reasonably assured there aren't any backdoors

Many people can compile code from source. Not so many people have the ability to audit code for obscured backdoors. The number of people that are capable to audit and have the time that is necessary to do it is practically nil.