Tools like bindiff have been around for years and take advantage of the fact that compilers don't randomize code generation. Instead, the callgraph and control-flow graphs largely reflect the structure of the original source code. Once you have leverage by exact-matching the parts of the binary that are nearly identical, you can build up and down the tree of nodes to find those that have more changes.
Crypto backdoors can be unbelievably subtle though. A single branch condition, a bit that is flipped, etc. can all lead to catastrophic failures. For example, a compiler optimization for dead code elimination led to some zeroization of key material being skipped. This kind of thing is extremely difficult to find and requires a careful understanding of the underlying code.
I agree with you that most differences can be found, but understanding the ramifications of those differences requires extremely careful analysis. A crypto flaw does not stand out from a mis-optimization.
That sounds like a pretty broken compiler, do you have a minimal example?
void
encrypt(void *data, size_t len, char *password)
{
char key[32];
turn_password_into_key(password, key, sizeof key);
aes_make_encrypted(data, len, key);
memset(key, 0, sizeof key); /* optimized away */
}
The compiler knows what memset does. It also knows that stack variables have no use after the function returns. Therefore, the compiler knows there is no reason to write zeroes to this memory, because the program will never read those zeroes. Hence, the compiler will delete the call to memset.Backdoors in source can be as simple as changing a single == to = or removing a minus sign in some seemingly innocuous place.
However there are plenty of examples of compilers aggressively removing code that causes undefined behaviour. Basically, when the compiler encounters UB, it can do whatever it wants to code that triggers the UB, which means possibly removing it. Compilers exploit this fact a lot because UBs happen a lot in "normal code". See here [2] for examples and explanations; I also can't help but mention John Regehr's blog [3] if you're interested in compilers, security, testing, safety.
[1] http://research.swtch.com/openssl
[2] http://blog.llvm.org/2011/05/what-every-c-programmer-should-...
[3] http://blog.regehr.org/archives/213 is a great article for example.
Why do you say that? Compiling the source with the same compiler and flags, plus diffing the binaries would quickly show where the differences lie, and if they're hostile. Any half-decent reverse engineer could do this.
That would stand out more than if the source itself was backdoored in a non-glaring way. Open source has taught us that nobody ever reads the source.